trk7's blog
type: channel
date: 2026-03-31T02:52:04Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Kubernetes Probes Are Not Your Dependency Strategy
items: How to Find What’s Driving BigQuery Bytes Billed
items: AI agents for coding: the promise, the potholes, and what’s next
items: Story & Technique
items: AI Agents for Coding: Promise vs. Reality
items: Looking beyond JSON Output format from LLMs
items: My Secret Weapon for Selective Exam Writing Practice
items: Upgrading Nginx Ingress Controller from V1.1.1 to V1.12.0
items: Going Live on Google Cloud Marketplace
items: Gandalf AI – Fun Game

Kubernetes Probes Are Not Your Dependency Strategy
type: item
creator: k7
date: 2026-03-31T02:52:03Z
subject: SRE
subject: k8s
description: Designing failure semantics across ingress, Java services, clients, and serverless The most dangerous thing about Kubernetes probes is not that teams misunderstand their definitions. It is that teams forget probes are actions, not just signals. A failed readiness probe changes routing. A failed liveness probe kills a process. A failed startup probe can stall a […] The post Kubernetes Probes Are Not Your Dependency Strategy appeared first on trk7's blog.
encoded: Designing failure semantics across ingress, Java services, clients, and serverless The most dangerous thing about Kubernetes probes is not that teams misunderstand their definitions. It is that teams forget probes are actions, not just signals. A failed readiness probe changes routing. A failed liveness probe kills a process. A failed startup probe can stall a rollout. That means probes do not merely detect failure; they shape it. Kubernetes is explicit here: startup gates the other probes, readiness controls whether traffic is sent to a pod, and liveness exists for restart-worthy states such as deadlock. It also warns that incorrectly implemented liveness probes can cause cascading failures. That is the lens required for a modern Kubernetes architecture: Client → NGINX Ingress → API service → backend service → managed SQL + event bus + internal services + LLM APIs The real design question is not “what is liveness vs. readiness?” It is: When each dependency fails, what behavior do we want from the platform, the service, and the client? What the Public Incidents Actually Teach Theory is great, but post-mortems are where we learn how systems actually die. Two major public outages serve as masterclasses in why tools designed to protect systems can weaponize a minor glitch into a catastrophic outage if they aren’t layered correctly. 1. GitHub’s August 2024 Outage: The Brittleness of Synthetic Health Checks In August 2024, GitHub suffered a massive outage affecting core read services. The root cause was not a database crash. Instead, a configuration change inadvertently broke the synthetic health-check pings used by their database routing layer (the proxies managing the MySQL replicas). Because the proxies couldn’t get a successful ping, they marked perfectly capable database nodes as DOWN and stopped sending them traffic. The Lesson: This highlights the profound danger of giving automation the power to sever traffic based on a synthetic check. The database was likely st…

How to Find What’s Driving BigQuery Bytes Billed
type: item
creator: k7
date: 2026-03-27T10:17:38Z
subject: Cloud
subject: GCP
subject: Uncategorized
description: Recently, I reduced BigQuery cost significantly by changing how we maintained a token index table used for keyword search. The original design updated the index during ingestion. Each small OCR batch triggered a MERGE into a chunk_terms table. That looked incremental, but it turned out to be expensive because BigQuery still needed to scan the […] The post How to Find What’s Driving BigQuery Bytes Billed appeared first on trk7's blog.
encoded: Recently, I reduced BigQuery cost significantly by changing how we maintained a token index table used for keyword search. The original design updated the index during ingestion. Each small OCR batch triggered a MERGE into a chunk_terms table. That looked incremental, but it turned out to be expensive because BigQuery still needed to scan the target table on every MERGE to check for matches. Since ingestion happened in many small batches, the same table was scanned repeatedly, which pushed bytes billed much higher than expected. The fix was: stop updating chunk_terms during ingest write only the primary records during ingestion move token indexing to a scheduled Cloud Run backfill job process recent or missing documents in batches scope each MERGE tightly and dedupe terms before insert The key lesson is simple: in BigQuery, workload shape matters. A workload made of many tiny MERGEs can cost much more than one built from fewer batched operations, even if the logical output is the same. To see where BigQuery spend is really going, INFORMATION_SCHEMA.JOBS_BY_PROJECT is extremely useful. These are the first queries I run. 1) Top expensive jobs in the last 7 days Use this to find the biggest individual jobs by bytes billed. SELECT creation_time, job_id, user_email, statement_type, total_bytes_billed, total_bytes_processed, query FROM `PROJECT_ID.region-us-central1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) AND job_type = 'QUERY' AND total_bytes_billed IS NOT NULL ORDER BY total_bytes_billed DESC LIMIT 50; 2) Bytes billed by statement type This helps separate spend across SELECT, MERGE, UPDATE, DELETE, and INSERT. SELECT statement_type, COUNT(*) AS jobs, SUM(total_bytes_billed) AS bytes_billed, SUM(total_bytes_processed) AS bytes_processed FROM `PROJECT_ID.region-us-central1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) AND job_type = 'QUERY'…

AI agents for coding: the promise, the potholes, and what’s next
type: item
creator: k7
date: 2025-09-21T10:12:57Z
subject: LLM
description: After publishing my last post on AI coding agents, I read Will Lockett’s piece, “The AI bubble is about to burst (but the next bubble is already growing)”. It nudged me to write a follow-up—not because I think AI is doomed, but because the gap between promise and day-to-day reality is still wide for working […] The post AI agents for coding: the promise, the potholes, and what’s next appeared first on trk7's blog.
encoded: After publishing my last post on AI coding agents, I read Will Lockett’s piece, “The AI bubble is about to burst (but the next bubble is already growing)”. It nudged me to write a follow-up—not because I think AI is doomed, but because the gap between promise and day-to-day reality is still wide for working engineers. Since that last article, I’ve spent more time with Google Code Assist, Codex, Jules, and Aider across a few backend LLMs. I’ve also reviewed and maintained AI-generated code in real projects. My current feeling: these tools can be brilliant accelerants up to a point—and then they start introducing friction that’s hard to see until you’re deep in it. A small saga: the two-day rollback One recent example: AI-generated code set a custom Postgres schema for Alembic—both for the version table and application tables. For days I couldn’t figure out why migrations weren’t taking effect. The commands ran cleanly, no errors, but every migration ended with a mysterious ROLLBACK in the emitted SQL. Multiple LLMs calmly told me: “That’s normal.” It wasn’t. After a lot of back-and-forth, I discovered the root cause: our SQLAlchemy connections were wrapping Alembic’s transactions; when the connection closed, SQLAlchemy issued a rollback that also rolled back Alembic’s work. The kicker? An AI finally helped me confirm the exact sequence of events. So yes—AI helped fix a bug that AI had introduced. It just cost two intense days to get there. This is not an isolated story. The ecosystem is moving at breakneck speed—Agentic frameworks, LLM-powered apps, RAG, MCP, A2A—and the hype cycle keeps compressing the time we get to actually master any of it. The tools and the expectations feed each other. How my metrics look now I’m still tracking a few dimensions to keep myself honest: Context retention Slight improvement, still brittle at the wrong moments. When it fails, it fails loudly. Strategic thinking Better. I see more plausible high-level plans and occasionally learn fro…

Story & Technique
type: item
creator: k7
date: 2025-07-16T11:52:51Z
subject: cinema
description: Recently I watched two movies that I was quite impressed with. October Sky A boy from a small town becomes inspired by the launch of Sputnik, prompting him to build amateur rockets with his friends and, eventually, work at NASA. I especially liked the dialogue between the father and son, in which the son tries […] The post Story & Technique appeared first on trk7's blog.
encoded: Recently I watched two movies that I was quite impressed with. October Sky A boy from a small town becomes inspired by the launch of Sputnik, prompting him to build amateur rockets with his friends and, eventually, work at NASA. I especially liked the dialogue between the father and son, in which the son tries to convince his father that he is confusing his own dreams with his son’s. Of course, the father encourages his son to pursue a future that, according to him, is the best possible option–but the son has his own dreams. Sometimes it is hard to see beyond what we know; a lack of imagination and ego can come into play when dealing with one’s children despite best intention. I also liked the teacher character, who works hard to inspire the kids to think beyond life working at a coal mine. The film made me realise that many people contribute to an individual’s success. Layer Cake This movie, on the other hand, was a photographic masterpiece–at least according to me. I liked the long, expansive shot of the park. I loved how they covered the massive warehouse at the end. I adored the sequence in which the main character was watching another man climb out of the pool and shoot him; as the man fell, the sprinklers in the backyard turned on and then morphed into a showerhead above the main character–who is now in the water. Pool → House, Sprinkler → Showerhead. The whole thing just clicked for me. The post Story & Technique appeared first on trk7's blog.

AI Agents for Coding: Promise vs. Reality
type: item
creator: k7
date: 2025-06-05T12:31:47Z
subject: programming
subject: software
subject: agents
description: Introduction As a software and infrastructure engineer with decades of experience, I’ve witnessed numerous technological shifts in our industry. Today, I’m exploring what might be another pivotal moment: the rise of AI coding agents. I use various AI tools daily (Perplexity, Repl.it, Gemini app, Claude API, GPT API, Ollama, Tabnine) for different tasks, but recently […] The post AI Agents for Coding: Promise vs. Reality appeared first on trk7's blog.
encoded: Introduction As a software and infrastructure engineer with decades of experience, I’ve witnessed numerous technological shifts in our industry. Today, I’m exploring what might be another pivotal moment: the rise of AI coding agents. I use various AI tools daily (Perplexity, Repl.it, Gemini app, Claude API, GPT API, Ollama, Tabnine) for different tasks, but recently decided to test their capabilities on a real-world engineering challenge. The Objective My goal was straightforward but non-trivial: fix a broken Firebase Cloud Messaging (FCM) notification system in a Laravel 5.5 application. The existing package had stopped working because it relied on an API version no longer supported, and the package itself was unmaintained. I had two potential solutions: Upgrade the entire Laravel application to a newer version with compatible FCM packages Replace just the FCM functionality with a custom implementation using the new API The Approach I decided to use AI agents to solve this problem, starting with the first approach – a full Laravel upgrade. This seemed like an ideal test case for agentic capabilities: Well-defined upgrade path (5.5 → 5.6 → 5.7 → 6.0 → 7.0 → 8.0 → 9.0 → 10.0 → 11.0 → 12.0) Clear documentation available Repetitive but detail-oriented work Required careful planning and execution The Agent Experience Attempt 1: Full Laravel Upgrade I asked Cursor to upgrade my application from Laravel 5.5 to 12.0. Despite multiple attempts and guidance, the agent: Repeatedly tried and failed with the same approaches Forgot critical context (like the location of docker-compose.yml) Made inconsistent changes to package versions Failed to maintain a coherent upgrade strategy After several frustrating iterations, I abandoned this approach. Attempt 2: Targeted FCM Solution I pivoted to asking Windsurf SWE to create a class wrapping the new FCM API. This was successful: It generated a well-structured class I was able to create a Service Provider and additional wrappers Initia…

Looking beyond JSON Output format from LLMs
type: item
creator: k7
date: 2025-05-24T06:03:46Z
subject: LLM
subject: quickfixes
subject: snippets
description: While working on my site for Learning and Quiz Generation I noticed that asking LLM’s for JSON outputs was problematic. Too many times the output was not a valid JSON. After multiple such issues, I felt that I should like at other output formats that I can then parse. Once such method that seems to […] The post Looking beyond JSON Output format from LLMs appeared first on trk7's blog.
encoded: While working on my site for Learning and Quiz Generation I noticed that asking LLM’s for JSON outputs was problematic. Too many times the output was not a valid JSON. After multiple such issues, I felt that I should like at other output formats that I can then parse. Once such method that seems to work is make use of Tag Markers. So instead of asking the LLM to output responses in this format: "questions": [ { "text" : "Full text/context that will be used by all subquestions", "subquestions" : [ "question": "Clear and detailed question.", "options": ["Option 1", "Option 2", "Option 3", "Option 4"], "correctAnswer": "Exact text of the correct option", "explanation": "Detailed explanation of why this answer is correct", "difficulty": number between 1 and 100, "topics": ["Topic 1", "Topic 2"] ] } ] I asked them generate response in this expected format: <QUESTION_START> <TEXT>Full text/context that will be used by all subquestions</TEXT> <SUBQUESTION_START> <QUESTION>First subquestion - clear and detailed question.</QUESTION> <OPTION_1>First option</OPTION_1> <OPTION_2>Second option</OPTION_2> <OPTION_3>Third option</OPTION_3> <OPTION_4>Fourth option</OPTION_4> <CORRECT_ANSWER>Exact text of the correct option</CORRECT_ANSWER> <EXPLANATION>Detailed explanation of why this answer is correct</EXPLANATION> <DIFFICULTY>number between 1 and 100</DIFFICULTY> <TOPICS>Topic 1, Topic 2</TOPICS> <SUBQUESTION_END> </QUESTION_END> And used a Regex like this to extract the text between the Tag Markers: <TAG_MARKER>([\s\S]*?)<\/TAG_MARKER> Seems to work really well so far. The post Looking beyond JSON Output format from LLMs appeared first on trk7's blog.

My Secret Weapon for Selective Exam Writing Practice
type: item
creator: k7
date: 2025-05-11T11:05:16Z
subject: my_sites
description: That Selective Exam Writing Test… Phew! Here’s Something I Built That’s Really Helping My Kid (And Might Help Yours Too!) If you know what I mean already, head over to the site now to register: https://bussin.trk7.app/ So, your child is gearing up for the Selective Exam? If your household is anything like mine, you know […] The post My Secret Weapon for Selective Exam Writing Practice appeared first on trk7's blog.
encoded: That Selective Exam Writing Test… Phew! Here’s Something I Built That’s Really Helping My Kid (And Might Help Yours Too!) If you know what I mean already, head over to the site now to register: https://bussin.trk7.app/ So, your child is gearing up for the Selective Exam? If your household is anything like mine, you know that the writing section can feel like a bit of a mountain to climb. I was definitely feeling the pressure for my own child, wanting them to walk into that exam feeling confident and prepared. Honestly, I looked around and couldn’t find exactly what I pictured to help them practice effectively. So, what’s a slightly tech-inclined parent to do? I decided to build it myself! A couple of times a week, I get my child to hop onto this website I created. It throws them a random writing topic – the kind of thing they’ll actually see in the real test – and they get about 30 minutes to write, just like the exam conditions. It’s been a good way to get them used to thinking on their feet. The really cool part, I think, is what happens after they submit their piece. It’s not just about getting a score. The site gives them a proper breakdown – looking at their creativity, how their grammar and spelling are holding up, the vocabulary they’re using, and all that good stuff. Plus, and this was super important to me, it offers real, practical examples of what they can do to make their writing even better next time. No vague comments, just actual usable tips. I’ve been tinkering with this and building it up over the last few months. It started as just a little something for us, but after running through over 50 practice tests and seeing the difference it’s making, I thought, “Hey, maybe other parents are in the same boat and could find this useful too?” I genuinely believe this could be a real help for your child. It’s not some big corporate thing; it’s literally something I built in my own time because I wanted to help my kid, and now hopefully, yours. If you’re curi…

Upgrading Nginx Ingress Controller from V1.1.1 to V1.12.0
type: item
creator: k7
date: 2025-02-07T21:43:28Z
subject: quickfixes
subject: sysadmin
subject: ingress
subject: kubernetes
subject: nginx
description: While upgrading the Nginx Ingress Controller, I encountered a few issues with the controller pod startup. Here are the problems and their solutions: 1. Cross-namespace secret is not supported If you have your mTLS CA as a secret in the namespace where the controller is deployed, you’ll face this issue. After the upgrade, the ingress […] The post Upgrading Nginx Ingress Controller from V1.1.1 to V1.12.0 appeared first on trk7's blog.
encoded: While upgrading the Nginx Ingress Controller, I encountered a few issues with the controller pod startup. Here are the problems and their solutions: 1. Cross-namespace secret is not supported If you have your mTLS CA as a secret in the namespace where the controller is deployed, you’ll face this issue. After the upgrade, the ingress deployed in the application’s namespace will not have access to this secret by default. Solution: Add the following configmap value: allow-cross-namespace-resources: "true" 2. Config file in /tmp/nginx/nginxcfg* does not exist The new version uses /tmp/nginx instead of /tmp. Solution: Create a volume mount with the mount path /tmp/nginx. These solutions should help you smoothly upgrade your Nginx Ingress Controller from V1.1.1 to V1.12.0. The post Upgrading Nginx Ingress Controller from V1.1.1 to V1.12.0 appeared first on trk7's blog.

Going Live on Google Cloud Marketplace
type: item
creator: k7
date: 2024-12-01T06:04:26Z
subject: GCP
subject: Uncategorized
subject: marketplace
subject: mogoplus
description: A Technical Lead’s Experience As the technical lead for MOGOPLUS’s recent launch on Google Cloud Marketplace, I want to share our journey and key insights. MOGOPLUS provides insights from unstructured data, primarily serving the lending sector. Our decision to join the Google Cloud Marketplace stemmed from our existing relationship as a GCP customer and our […] The post Going Live on Google Cloud Marketplace appeared first on trk7's blog.
encoded: A Technical Lead’s Experience As the technical lead for MOGOPLUS’s recent launch on Google Cloud Marketplace, I want to share our journey and key insights. MOGOPLUS provides insights from unstructured data, primarily serving the lending sector. Our decision to join the Google Cloud Marketplace stemmed from our existing relationship as a GCP customer and our desire to expand our collaboration with Google. Understanding Google Cloud Partnership Google Cloud offers three partnership models: Sell Partner: Resell Google Cloud products Service Partner: Provide consulting and service delivery Build Partner: Develop integrated solutions for Google Cloud MOGOPLUS chose the Build Partner route to sell our service on the Google Cloud Marketplace. The Journey to Marketplace: Key Steps While the complete process involves six steps, I’ll focus on the technical aspects of steps 4-6. For a full overview of the process, refer to the official Google Cloud documentation. 1. Create a Technical Diagram Develop a high-level diagram showcasing your solution’s architecture Highlight the Google Cloud services your solution utilizes Keep it simple yet informative For inspiration, check out these Google Cloud reference architectures 2. Prepare Documentation Describe your solution, the problem it solves, and its industry relevance Explain how the technical solution meets business requirements 3. Estimate GCP Usage Use the Google Cloud Pricing Calculator Work with your business team to project revenue and costs Ensure a reasonable ratio between projected revenue and GCP costs 4. Set Up GCP Projects and IAM Create separate projects for marketplace solutions Set up appropriate permissions for Google service accounts Consider creating non-prod environments for testing Follow the official setup guide 5. Define Pricing Model Collaborate with the commercial team on pricing strategy Determine metrics for usage-based pricing (if applicable) Review pricing options 6. Develop Product Listing Work with Pr…

Gandalf AI – Fun Game
type: item
creator: k7
date: 2024-11-10T08:45:17Z
subject: Machine Learning
subject: game
subject: LLM
description: I had a lot of fun fun while playing this game. The objective is for the player to extract password from the system. The first few levels where straightforward and the next few I realised at the end that I could have used almost the same prompt 🙂 For the last one, I had to […] The post Gandalf AI – Fun Game appeared first on trk7's blog.
encoded: I had a lot of fun fun while playing this game. The objective is for the player to extract password from the system. The first few levels where straightforward and the next few I realised at the end that I could have used almost the same prompt For the last one, I had to do some extra trickery. Hint: You need get it to reveal it’s prompt but without tripping the verification agent that checks the response before presenting to you. While playing it I realised that the AI systems that we have in the open now are both smart and dumb at the same time. Have fun <img fetchpriority="high" decoding="async" width="554" height="1024" data-id="522" src="https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf-554x1024.png" alt="" class="wp-image-522" srcset="https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf-554x1024.png 554w, https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf-162x300.png 162w, https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf-768x1419.png 768w, https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf-832x1536.png 832w, https://www.trk7.com/blog/wp-content/uploads/2024/11/gandalf.png 935w" sizes="(max-width: 554px) 100vw, 554px" /> The post Gandalf AI – Fun Game appeared first on trk7's blog.