View Details

While AI coding assistants have become mainstream, applying generative AI to infrastructure and operations is still emerging. The promise is huge—but so are the challenges, from hallucinated insights to navigating noisy telemetry.

In this episode of Semaphore Uncut, Gou Rao—AI infrastructure veteran and co-founder of Neubird—joins Darko to explore how LLMs can support DevOps teams not just with suggestions, but with reasoning, context gathering, and real-time incident diagnosis. He shares the vision behind Hawkeye, Neubird’s “digital engineer,” and what it means to build agentic systems that think and operate like humans.

A Career Rooted in Data and AIGou’s background in AI and enterprise systems goes back to his graduate work at the University of Pennsylvania. Over the years, he’s applied data science to solve tough infrastructure problems, most recently as part of the founding team at Portworx, a cloud-native storage platform.

At Neubird, Gou brings together his two longtime passions: data structures and AI. His team is now focused on building tools that use LLMs not just to generate content, but to extract operational insights from sprawling, high-volume telemetry systems.

From Stack Traces to Reasoning EnginesMost engineers today interact with LLMs in a straightforward way: paste in a stack trace, ask for help, and get a response. But Neubird’s thesis is different.

“The hard part isn’t solving the stack trace,” Gou says. “It’s finding it.”

Neubird’s product, Hawkeye, acts as a reasoning engine across the entire telemetry stack—logs, metrics, traces, configuration, alerts. Rather than using retrieval-augmented generation, it reverses the pattern: LLMs decide what needs to be found, and the system fetches it through native queries (like PromQL or Splunk searches). That separation improves both accuracy and auditability.

Building a Digital EngineerThe team designed Hawkeye to mirror how a human SRE would think: identify symptoms, investigate telemetry sources, reason about possible causes, and suggest remediations. Behind the scenes, different models (Claude, Mistral, Llama3) are orchestrated depending on task complexity.

Each inference takes a few minutes—not seconds—but that’s intentional. Speed isn’t the goal. Correctness is.

To build trust, Hawkeye shows its chain of thought as it works. For example, it might check a Prometheus metric, then search for recent config changes, then correlate that with a specific GitHub commit. While it doesn’t take action on its own yet, it can generate pull requests, annotate incident tickets, or suggest Terraform changes for review.

Telemetry-First, Not Model-FirstA major design principle behind Hawkeye is accuracy. Gou explains that they never rely solely on model output. If a model says, “Check if CPU usage is high,” the system runs an actual Prometheus query to verify that.

That tight integration with telemetry also means Hawkeye adapts to your environment. During onboarding, teams provide read-only access to their observability tools and infrastructure. From there, Hawkeye builds context, including preferences, frequently failing clusters, and past incidents stored in systems like ServiceNow.

While global model training happens based on aggregated feedback, environment-specific memory—like your preferred verbosity or known flaky services—is stored separately in a vector database to shape interactions without retraining the model.

Scaling from Startups to EnterprisesHawkeye is already proving useful across a wide customer base. One small company with a single SRE uses it to cut through noisy alerts and stay focused on critical issues. At the other end of the spectrum, large banks are integrating Hawkeye into high-stakes payment systems to augment their SRE teams and reduce stress and churn.

No matter the size, the goal is the same: reduce time spent sifting through logs and alerts, and make engineers more effective.

What About Autonomous Actions?Right now, Hawkeye only recommends actions—it doesn’t apply changes directly. That might change, but only with time and trust.

“There’s an unpredictability in generative AI that makes people cautious,” Gou says. “We’re not ready to go full self-driving in production. But maybe we’ll start with safe environments—nightly builds, staging systems, and so on.”

Like Tesla’s autopilot, Gou believes the future will include more automation, but only after the systems prove themselves repeatedly in controlled scenarios.

Trying Out HawkeyeInterested teams can try Hawkeye through Neubird’s website or directly via AWS and Azure marketplaces. A playground environment helps teams test key use cases before connecting real production data.

Once deployed, Hawkeye integrates seamlessly with observability platforms like Prometheus, Azure Monitor, or Splunk. Whether used in real-time triage or as an asynchronous assistant to incident response tools like PagerDuty or Jira, Hawkeye aims to be a thinking partner—not just another bot.

Final Thoughts: A Teammate, Not a ToolGou is clear-eyed about the limitations: Hawkeye isn’t magic. It doesn’t solve every incident or replace your ops team. But it augments engineers by handling the tedious groundwork—surfacing relevant signals, correlating issues, and providing well-reasoned suggestions.

“It’s there to take the grunt work off your plate,” Gou says. “So you can focus on the interesting stuff.”

And in a world where SREs are under constant pressure, that’s more than welcome.

Follow Gou Rao and Neubird🌐 Website: neubird.ai
💼 LinkedIn: Gou Rao | Hawkeye

The post Gou Rao on Agentic Systems in DevOps appeared first on Semaphore.

View Details

We’re excited to share what’s new in Semaphore v1.3 — a release focused on empowering contributors, improving the local development experience, and making advanced CI/CD features more accessible to small teams.

Whether you’re looking to make your first contribution or run Semaphore in your organization, this update brings significant improvements across the board.

Contributing Just Got a Lot EasierSince open-sourcing Semaphore, we’ve welcomed contributions, but we knew the developer experience needed refinement. With v1.3, we’ve made that process much smoother:

• Forked PRs now trigger CI automaticallyNo more waiting for manual approvals from our team to run tests. As soon as you open a pull request, CI/CD kicks in.
• Public access to pipeline jobsYou can now inspect test results and logs from your fork, making debugging and iteration significantly easier.
• Improved contributor documentationOur DEVELOPMENT.md and CONTRIBUTING.md files now explain everything from tooling to branch naming conventions and commit message formatting.
• Local development with ScaffoldYou can now run Semaphore locally using Skaffold. We’ve consolidated our microservices to reduce the resource footprint to around 7GB RAM and 5 CPUs — and we’re actively working on lowering it even more.

First-Time Contributors SpotlightWe’re thrilled to welcome two new external contributors:

• Michał Matyas fixed a timestamp formatting issue in the activity monitor UI.
• Fred resolved a YAML switching bug in our visual editor and fixed typos in the getting started guide.

Thank you both! 🙌

You can find more opportunities labeled as good first issue on our GitHub board. These are ideal if you’re looking to get familiar with our codebase.

Better Workflow Navigation & FilteringWe’ve improved how you browse workflow history:

• Stateful paginationThe URL now retains your place when navigating between pages or sharing links.
• Date-based filteringYou can filter workflows by specific date ranges — helpful when tracking down long-running or historical builds.

Announcing the Semaphore Enterprise Edition — Free for Small TeamsWe’re introducing a new Enterprise Edition of Semaphore. And it’s free for small teams!

If your company has:

• Fewer than 50 employees
• Less than $5M in annual recurring revenue

You can use the full power of Semaphore, including advanced RBAC and compliance features, at no cost. Just install the Enterprise edition using Helm and request a free license key. More details in our Enterprise Edition docs.

🐛 Coming Soon: Bug Bounties via AlgoraWe’re also launching a bug bounty program in partnership with Algora.io. You’ll soon be able to earn rewards for fixing issues and contributing features. Keep an eye on our GitHub and README for the official launch.

🔭 What’s Next in v1.4?Here’s a sneak peek of what’s coming:

• Partial pipeline rebuilds via the UIAlready available through our API, this long-requested feature is making its way into the UI.
• Further local dev optimizationsWe’re aiming to bring down CPU and memory requirements even more.
• More good-first-issuesExpect a growing list of approachable tasks for new contributors.

Thanks to everyone who contributed to this release — both inside and outside the Semaphore team. Version 1.3 is a major step toward making Semaphore easier to contribute to and more accessible to a wider community.

If you haven’t already, check out the repo, give us a ⭐, and let us know what you think.

Happy building!

The post Semaphore v1.3 Release Roundup: A Big Leap for Contributors and Small Teams 🚀 appeared first on Semaphore.

View Details

What really happens after you push code to your Git repository? Your pipeline kicks off, tests run, and builds are triggered. But, beneath the surface, a complex orchestration engine is making all the right moves.

At Semaphore, that engine is called Plumber.

Plumber is responsible for interpreting pipeline definitions, scheduling jobs, and coordinating execution across distributed agents — all while ensuring reliability and scalability. In this post, I’ll show you:

• How we built it using a finite state machine
• Why we chose Elixir
• How this architecture helps Semaphore scale CI/CD for thousands of developers.

Understanding Semaphore’s Pipeline ModelSemaphore pipelines are made up of blocks and jobs:

• A pipeline defines your CI/CD workflow.
• Blocks group jobs together and define order—sequential or parallel.
• Jobs are execution units that run commands.
• Agents are the machines that actually execute the jobs.

This structure allows teams to model anything from basic test pipelines to complex multi-stage deployments.

From Git Push to Pipeline ExecutionWhen you push code:

1. Your Git provider sends a webhook to Semaphore.
2. The Hooks Processor validates and transforms the event.
3. The Pipeline Processor—this is where Plumber comes in—fetches the YAML definition and starts orchestration.
4. The Job Processor prepares and assigns jobs to available agents.
5. Agents execute jobs and report results back to the platform.

Plumber is the service that turns a definition into a real, running pipeline.

Why a State Machine?We designed Plumber around a finite state machine (FSM) model. Each pipeline transitions through a known set of states:

• initializing: YAML is parsed and blocks/jobs are created.
• pending: Conditions (like branch filters) are evaluated.
• running: Jobs are executed in the correct order.
• done: The result is finalized and published.

Each state is handled by a dedicated Elixir Looper process that executes state-specific logic. This keeps our system modular, predictable, and easy to maintain.

The Tech StackHere’s what powers Plumber:

• Elixir: Built for concurrency, fault tolerance, and lightweight processes.
• PostgreSQL: Provides transactional integrity for all state transitions.
• RabbitMQ: Ensures reliable communication between services.
• Kubernetes: Handles deployment, scaling, and recovery.

This stack lets us scale horizontally, recover from failure, and roll out changes with confidence.

Handling Failures GracefullyThings can go wrong. A worker might crash mid-job, or an external dependency might fail. To prevent stuck pipelines, we added a stuck state and a recovery loop.

A dedicated recovery worker detects pipelines that have stalled, retries them with exponential backoff, and uses a circuit breaker to stop retrying if the issue persists. This allows the system to self-heal while keeping other pipelines unaffected.

Extending Pipeline BehaviorThanks to the FSM model, adding new features is straightforward.

Stopping Pipelines ManuallyWhen a user stops a pipeline mid-run, we transition it into a stopping state. All running jobs are halted before the pipeline is done.

Queuing PipelinesIf multiple commits are pushed to the same branch in a short period, they shouldn’t run in parallel. We added a queuing state so that newer pipelines wait for earlier ones to finish.

Recap: Why It WorksPlumber’s architecture supports the demands of modern CI/CD by focusing on a few key principles:

• Reliability: Supervision trees and Kubernetes health checks keep workers available.
• Scalability: Looper workers scale horizontally across pods.
• Fault Tolerance: Stuck state recovery prevents cascading failures.
• Extensibility: New pipeline logic fits naturally into the state machine.

It’s a system designed to grow with developers’ needs, and it’s been proven in production every day.

Learn MoreSemaphore is open source. You can check out the code and dive deeper into how it works:

🔗 Explore it on GitHub
💬 Join our Discord to ask questions (will add link)
📺 Watch related talks on our YouTube channel

Happy building!

The post The Secrets Behind Plumber: Managing Thousands of Pipelines at Semaphore appeared first on Semaphore.

View Details

Since we open sourced Semaphore with the Community Edition, I wanted to make a detailed comparison with Jenkins, the most popular and well-known open-source CI platform – by far. In this post, I’ll break down the differences in licensing, hardware requirements, installation, access control, redundancy, features, scalability, performance, and maintenance.

By the end, you’ll have a clear understanding of the strengths and weaknesses of each platform, helping you decide which one aligns best with your project needs.

Comparison CategoriesBefore starting, let’s break down the categories we’re going to use in this comparison.

| Feature | Jenkins | Semaphore CE | | --- | --- | --- | | License | MIT | Apache-2.0 | | Hardware Requirements | 4 GB RAM, 1 CPU | 16 GB RAM, 8 CPUs | | Installation | Docker or standalone service | Kubernetes and Helm | | Access Control | Plugin-based | Built-in RBAC | | Redundancy | Active-Passive Setup | Kubernetes-native | | Monorepo Support | Only via plugins | Natively suported | | Test Reporting | Built-in | Built-in Dashboard | | Notifications | Plugins required | Built-in Webhooks | | Scalability | via Worker Nodes | Kubernetes native scalability and self-hosted agents | | Parallelization | Must be specifically configured | Default | | Maintenance | High (Plugin-Dependent) | Low (Minimal Setup) |

License ComparisonJenkins is licensed under the MIT License, making it highly permissive for modification and redistribution.

Semaphore CE, on the other hand, is licensed under Apache-2.0, a similarly permissive license that also includes protections against patent litigation.

| Aspect | MIT License | Apache-2.0 License | | --- | --- | --- | | Permissions | Permissive, allows modification and redistribution | Permissive, allows modification and redistribution | | Conditions | Requires inclusion of the original license | Requires inclusion of the original license, NOTICE file, and state of modifications | | Limitations | No liability or warranty | No liability or warranty | | Patent Protection | No patent protection | Includes explicit patent protection | | Sublicensing | Allowed | Allowed | | Attribution | Required | Required |

Scalability and PerformanceSemaphore CE was designed as a cloud-native platform, leveraging Kubernetes and self-hosted agents for distributed workloads.

Jenkins scales using additional nodes, but requires more manual setup and maintenance.

Semaphore CE’s autoscaler further simplifies scaling for large workloads on demand when using AWS for compute.

ParallelizationJenkins supports parallelization using the parallel keyword in its Jenkinsfile, but all jobs run in the same environment, potentially causing race conditions.

Semaphore CE runs each job in a separate Docker container (or VM) by default, enabling easier and more reliable parallel execution.

Hardware RequirementsJenkins has relatively modest hardware requirements, recommending at least 4 GB of RAM and 1 CPU, with scaling recommendations based on job count and agent connections (1 CPU per 100 jobs).

Semaphore CE, designed to run on Kubernetes, has higher minimum requirements with at least 16 GB RAM and 8 CPUs for the control plane, making it more resource-intensive but also more powerful in terms of handling larger workloads.

InstallationJenkins can run fully inside a firewall and can be deployed using Docker, a standalone package, and run as a service.

In contrast, Semaphore CE has more requirements:

• a public IP address
• a domain
• a TLS certificate

This makes Semaphore’s setup slightly more involved but also more secure and modern in architecture.

Access ControlJenkins requires plugins for role-based access control, making it more customizable but also more complex to set up. Most Jenkins installations use the matrix authorization strategy plugin for this purpose.

Semaphore CE provides built-in role-based access control, allowing for straightforward user and permission management without the need for additional plugins.

RedundancyRedundancy in Jenkins is achieved using an active-passive strategy, where servers share a mount point, allowing one to take over if the other fails.

Semaphore CE leverages Kubernetes’ built-in redundancy, allowing for more robust failover and scaling capabilities.

FeaturesLet’s break down some features that I think we can’t live without in any software project.

Monorepo SupportJenkins can do monorepo if we configure multi-branch pipelines and use plugins, but setting this can be complex and feels brittle

Semaphore CE natively supports monorepos with the ‘change_in’ conditions, allowing for targeted job execution based on specific file changes, reducing unnecessary runs and optimizing resource usage.

Test ReportingBoth platforms support JUnit-based reports. Jenkins shows a success/failure graph over time and offers per-workflow reports.

In the meantime, Semaphore CE offers a built-in test dashboard with categorization, filtering, and test history tracking, making it more user-friendly and less reliant on plugins.

NotificationsJenkins requires plugins to send notifications to external services like Slack or Discord.

Semaphore CE includes built-in webhook support, enabling easier integration with popular communication tools without additional plugin installations.

MaintenanceJenkins’ plugin ecosystem can become a maintenance burden, requiring regular updates and conflict resolution.

Semaphore CE, by contrast, does not rely on any plugins to work, focusing on out-of-the-box functionality that reduces maintenance overhead.

ConclusionChoosing between Jenkins and Semaphore CE comes down to project size, team expertise, and infrastructure preferences. Jenkins offers greater flexibility and an extensive plugin ecosystem but demands more maintenance.

Semaphore CE provides a streamlined, Kubernetes-native experience, reducing setup time and operational complexity at the cost of higher hardware requirements.

Both platforms are open-source, so the best approach is to try both and evaluate based on specific project needs.

Thank you for reading, and happy building!

The post Semaphore Community Edition vs. Jenkins appeared first on Semaphore.

View Details

When we decided to open source Semaphore, we knew it wouldn’t be just about flipping a switch and making the repo public. A lot had to change, especially how we test and manage contributions.

Our CI/CD pipeline had been fine-tuned over the years to work with our internal infrastructure. But if we wanted to support real-world use cases—developers deploying Semaphore on Google Cloud, AWS, or even their own machines—we needed something new.

We needed to validate that every release of Semaphore would run smoothly anywhere. That meant spinning up fresh infrastructure, installing the app, running end-to-end tests, and tearing it all down.

So we built ephemeral environments. This is how we did it—and how it ended up being a lot harder than I thought.

The Goal: From Prod-Centric to PortableOur infrastructure had grown organically over a decade. Everything—our app config, testing setup, CI logic—was tailored to our exact environment.

Once we went open source, that was no longer acceptable. We had to prove that Semaphore could run cleanly in new environments, without the crutches of our internal setup.

So we set out to change the pipeline.

Here’s what the goal looked like:

1. Package the app using Helm charts.
2. Spin up a brand-new Kubernetes cluster.
3. Install Semaphore using those charts.
4. Run end-to-end tests.
5. Tear it all down when we’re done.

Seems reasonable, right?

First Stop: Google CloudSince our production runs on Google Cloud, starting with GKE made sense. We already used Terraform for our infrastructure, so we figured we could just repurpose what we had. But it turns our that wouldn’t work at all.

The main issue with Terraform is that our config had 270+ resources, including a bunch of networking and security stuff we didn’t need for the Community Edition. It was testing overkill.

So we started from scratch. The minimal version included:

• A VPC (with default subnets)
• A Kubernetes cluster
• A node pool
• A resource for storing TLS certificates
• A static IP

Just five resources. That was a huge win.

We hooked up a Helm chart to install Semaphore, passed in a domain name, and used a Google Load Balancer to expose it.

Then I hit my first real snag.

The Load Balancer ProblemSemaphore’s installer pings itself via the public domain to verify the install worked. But the load balancer doesn’t exist yet when installation starts. So, the installation fails.

The DNS needs to point to something. The solution was to reserve a static IP during provisioning, point the domain to that IP in DNS, and then make sure the load balancer uses that reserved IP once it spins up.

It worked. Not elegant, but functional.

From Manual to CIOnce we had the GCP-based ephemeral environment working manually,weI turned it into a CI workflow:

• Generate certs
• Provision infrastructure with Terraform
• Install Semaphore with Helm
• Run end-to-end tests
• Destroy everything

Each step runs in its own CI job, and we pass data between jobs using artifacts. The flow was finally shaping up.

Time to try the same on AWS.

Then Came EKSWe assumed this would be a “copy/paste and rename” kind of situation. But, yet again, we found a lot of unforeseen problems.

Here’s what broke immediately:

• Persistent volumes didn’t work. Unlike GCP, EKS doesn’t come with default storage classes.
• Pods didn’t schedule. Turns out, AWS limits pods per node based on IP allocation via ENIs. We had to split into multiple nodes.
• Load balancers didn’t auto-provision. GKE handles this for you. EKS doesn’t—unless you install a separate controller.
• No static IPs or DNS reservations. You only get the DNS name after the LB is created.

So we added:

• AWS Load Balancer Controller (LB), to actually spin up the LB from our Ingress definitions
• External DNS, to update our DNS zone automatically when the LB is ready

The extra complexity was painful, but it finally mirrored the GCP setup.

Meanwhile, Someone Just Used a VMWhile we were deep in Kubernetes land, a teammate spun up a minimal environment using a single VM and k3s.

It booted in 45 seconds. No load balancers. No controllers. No YAML debugging. It just ran.

We now use this approach for fast, local testing during feature development. When we need to validate against “real” infrastructure, we run the GCP and AWS ephemeral environments before a release.

What We Learned (the Hard Way)Here’s what we’d tell our past selves before starting the project:

1. Cloud services aren’t interchangeable. GCP and AWS might both offer “managed Kubernetes,” but the assumptions, defaults, and gotchas are wildly different.
2. Strip it down. Start with the simplest working Terraform you can. Add features later.
3. Expect weird infrastructure bugs. Load balancer DNS, pod scheduling limits, and missing storage classes—these took the most time to figure out.
4. Use automation to your advantage. CI pipelines, Terraform, Helm—once wired together, they save hours of repeat work.

Where We Are NowToday, every version of Semaphore goes through:

• Lightweight CI validation in a single-VM k3s setup
• Full end-to-end tests in GCP and AWS-based ephemeral environments

And everything is triggered automatically in CI.

What used to be a manual process tied to our infrastructure is now a portable, repeatable, and testable flow.

It wasn’t the easiest migration, but it was one of the most important things we’ve done for open-sourcing Semaphore.

If you’re working on something similar—or thinking about building ephemeral environments—feel free to drop by our Discord server or check out the repository. We’re always happy to chat.

The post Ephemeral Environments for Testing Open Source Projects appeared first on Semaphore.

View Details

In this episode of Semaphore Uncut, Darko Fabijan welcomes Marcelo Calbucci—engineer, startup founder, and author of The PR FAQ Book. Marcelo shares insights from his time at Amazon, where he encountered a decision-making framework that changed how he thinks about product development: the PR FAQ.

What starts as an internal press release quickly becomes a strategic tool to align teams, evaluate product ideas, and make better decisions—before writing a single line of code.

From Startups to Amazon—and Back AgainMarcelo’s journey spans startups, big tech, and now authorship. Originally from Brazil, he kicked off his career at Microsoft working on Bing, then founded multiple startups across Seattle and London. After 18 years in the startup world, he joined Amazon—a drastic shift in scale and structure.

That’s where he discovered the PR FAQ framework.

“I saw how useful it was, not just inside Amazon,” Marcelo says, “but for any founder, product leader, or engineer making strategic decisions.”

What Is a PR FAQ?PR FAQ stands for Press Release and Frequently Asked Questions. It’s more than a document—Marcelo describes it as a three-part system:

• A six-page narrative outlining the vision and FAQs
• A precise writing style that emphasizes clarity and data over buzzwords
• A method for team collaboration and decision-making

The structure is simple:

1. Page 1: A fictional press release, imagining the product is already launched
2. Page 2: Customer FAQs
3. Pages 3–6: Internal FAQs addressing strategy, feasibility, and impact

Unlike traditional planning documents, a PR FAQ avoids details about tech stacks, feature lists, or timelines. Instead, it focuses on why the project matters, what problem it solves, and who it serves.

Working Backwards: Strategy Before ExecutionAt the heart of the PR FAQ is a technique Amazon calls working backwards—starting with the customer-facing outcome and reverse-engineering the path to get there.

Marcelo explains:

“It’s a vision and strategy document. Not a plan. You’re imagining the future and asking: what must be true for this to succeed?”

By avoiding premature execution details, teams stay focused on the big picture. Including implementation specifics too early can derail strategic discussions.

The Lifecycle of a PR FAQA PR FAQ typically takes one to two weeks to develop. It begins with a draft from a project owner (like a CTO or product lead), followed by iterative review sessions with stakeholders. These reviews aren’t just approvals—they’re collaboration points to refine the vision.

Eventually, the document reaches a decision meeting. The result: go, no-go, or revisit later.

Once approved, the PR FAQ becomes the team’s north star. And unless a major strategic assumption changes, the document rarely needs rewriting.

“The value is clarity. Everyone knows why the project matters—and can make aligned decisions without constant check-ins.”

When Should You Use One?Marcelo emphasizes that PR FAQs aren’t only for product launches. You can use them for:

• Migrating vendors (e.g. switching observability tools)
• Strategic tech shifts (e.g. adopting Snowflake)
• Internal programs or process overhauls
• Even writing a book (Marcelo used one for his!)

The key isn’t the size of the task—it’s the impact. A one-day feature with major business implications might deserve a PR FAQ. A three-month refactor that introduces legal or compliance risks might need one too.

OKRs + PR FAQ = A Healthy StackMany teams use OKRs (Objectives and Key Results) for planning—but Marcelo warns that OKRs alone can lack context.

“We’d spend 10–15 minutes in meetings just explaining what the OKR meant. Eventually, we started writing a paragraph of narrative. At Amazon, I realized: that’s the PR FAQ.”

Marcelo sees PR FAQs as a perfect precursor to OKRs. They bring clarity before planning, and help teams avoid setting vague or misaligned objectives.

Common Pitfalls to AvoidMarcelo shares three mistakes teams often make when adopting PR FAQs:

1. Confusing strategy with execution
   Don’t list features or roadmaps. Stay focused on why.
2. Using vague or “salesy” language
   Be specific. Say “78% of users used this feature last week” instead of “most users love it.”
3. Treating it as a solo activity
   A PR FAQ isn’t a manifesto—it’s a collaborative document. Use it to pull your team in, not push ideas out.

Want to Try It?Marcelo’s book, The PR FAQ Book, is a practical guide filled with real examples, templates, and tips for using the framework in your own team.

🎁 Join the giveaway for Semaphore Uncut listeners: theprfaq.com/giveaway/semaphore-uncut

The post Marcelo Calbucci on Strategic Product Thinking appeared first on Semaphore.

View Details

What’s NewIn this release, we’ve added two major features:

• Markdown Reports: Render markdown files directly on the workflow and job pages.
• Git-Agnostic Integration: Connect any Git server via a simple push-hook script.

For more details on other changes, head over to the changelog.

📋 Markdown ReportsThe new Reports tab allows you to create rich, custom markdown reports to enhance and expand the UI of your job and workflow pages. This feature helps you add tailored visualizations, summaries, and detailed information without leaving Semaphore, making it easier to communicate build results and pipeline insights.

• New “Reports” tab: View .semaphore/REPORT.md files directly in the workflow or job pages, fully rendered for easy reading.
• Mermaid.js support: Visualize your pipeline with dynamic charts and diagrams using Mermaid.js.
• Easily generate reports: The test-results CLI has been expanded to help you generate basic job reports quickly.
• Fully customizable: Add test results, deployment notes, and other useful content to better organize and communicate your CI/CD process.

This feature helps teams present their pipeline results in a more meaningful way, making it easier to share insights and keep everyone on the same page.

🛠️ The test-results CLI UpdateThe test-results CLI has been updated with two new commands, making it even easier to generate detailed, insightful reports directly from your jobs.

• test-results command-metrics REPORT.md: Generates a timeline chart showing the execution time for each command in your job, and appends it to your report for a clear, visual representation of your job’s performance.
• test-results resource-metrics REPORT.md: For teams using Semaphore Cloud machines, this command pulls resource usage metrics (CPU, memory, disk) and displays them as neat charts in your report.

🔗 Git-Agnostic IntegrationSemaphore now supports integration with any Git server.

• What’s new: You can use a post-receive script to connect to any Git server, whether it’s a self-hosted Git instance, or another provider.
• How it works: After configuring the script, Semaphore can automatically trigger builds from pushes to your Git server — just like it does with GitHub, Bitbucket and GitLab.
• Why it matters: This adds flexibility, so teams using custom or self-hosted Git setups can use Semaphore without switching services.

This change removes any vendor lock-in and expands the range of Git providers you can work with.

👥 External Contributor ShoutoutWe’d like to give a special thanks to @colemickens from Determinate Systems for their contribution in extending and expanding our OIDC Token functionality. Your efforts helped make this release even better!

If you are a Nix fan, be sure to check out Determinate Systems.

🔮 What’s Coming in v1.3In the upcoming v1.3 release, we’ll be focusing on both improving the user experience and making it easier to contribute to Semaphore.

• Branch Page & Workflow List UX Improvements: We’ll be enhancing navigation and pagination, along with adding useful filters like a datetime filter to help you quickly find specific workflows from the past.
• Contributor Experience: We’re working on improving how contributors interact with Semaphore. This includes:
  • Better documentation of our architecture and services to help you understand how everything fits together.
  • Expanded contribution guide with more details on how to get involved.
  • Local testing solution: We’re investigating ways to allow you to test Semaphore changes locally before pushing them.

These updates aim to make Semaphore even easier to use and contribute to, whether you’re managing workflows or helping build the platform itself.

The post Semaphore v1.2 Release Roundup: Reports and Git Agnostic appeared first on Semaphore.

View Details

At Semaphore, we’ve been rethinking our public API from the ground up.

Over the years, as our platform grew, so did the complexity of our API infrastructure. We tried multiple approaches—some promising, others not so much. The result? A fragmented developer experience that made it harder to integrate, extend, and maintain.

So we decided to change that.

This post is about our journey to unify the Semaphore API behind a single, resource-oriented design. It’s about making things simpler, more consistent, and easier to use—not just for us, but for every developer who builds on Semaphore.

Where Things Got MessyOur API has evolved in layers. At one point or another, we were running:

• gRPC APIs that exposed through a reverse proxy, translating HTTP to gRPC.
• Per-service HTTP APIs which were maintained independently.
• A dedicated HTTP server translating requests into internal service calls.

Each method solved problems in its own silo. But together? They made our system hard to reason about and even harder to scale.

The Turning Point: A Dedicated API GatewayThe most promising approach was the dedicated HTTP gateway: a single service that receives external HTTP requests and translates them into internal gRPC calls.

This gave us several key advantages:

• A single point for applying authentication, validation, and formatting
• One source of truth for the request/response payloads
• Easier maintenance as new features are added
• Consistency in how internal and public APIs communicate

It was such an obvious choice that we leaned in.

Resource-Oriented API DesignTo make the API intuitive and extendable, we adopted a resource-oriented design. Inspired by Kubernetes and REST best practices, our resources now follow a consistent structure:

{ "kind": "project", "metadata": { "id": "proj-123" }, "spec": { "name": "My App", "description": "CI/CD pipeline" }} * kind tells you what type of resource you’re working with. * metadata includes identifiers and timestamps. * spec holds the actual configuration or desired state.

This pattern makes it easier to reason about objects across the system and gives us room to grow the API without breaking clients.

Richer RepresentationsAnother upgrade we made during design is to include rich object representations of objects instead of their raw IDs.

So, for example, instead of:

"created_by": "user-456" You now get:

"created_by": { "kind": "user", "id": "user-456", "name": "amir"} This gives consumers more context without extra API calls and sets us up for future extensibility.

Supporting Actions Beyond CRUDNot all operations fit neatly into create/read/update/delete actions. For example, starting a workflow requires more than just updating a resource.

So we introduced custom methods for resource-specific actions:

POST /projects/{project_id}/workflows/{id}/start These methods are scoped to the resource, follow predictable naming conventions, and include actionable error messages when things go wrong.

Built with OpenAPI (and Built to Last)A major part of this overhaul was committing to OpenAPI-first development.

Each endpoint in the API is defined with:

• Typed request and response schemas
• Collocated OpenAPI specs
• Automatic validation and permission checks
• Clear documentation generated from the source

This keeps our docs accurate, reduces drift, and makes the API easier to contribute to and extend.

Inside the Project StructureHere’s how everything fits together under the one API:

• Handlers: we use one file per endpoint. Each files handles casting, validation, logging, permissions, and forwarding requests.
• Internal clients: we use them as bridges between handlers and gRPC services.
• Formatters: we use them to translate between internal requests and external responses.

We’ve designed things to be modular and clear, so contributors can get up to speed quickly. It’s now easier than ever to add a new endpoint or update an existing one.

Want to Get Involved?The Semaphore API is open source, and we’d love your feedback.

If you’re a developer who:

• Wants to help improve a real-world API
• Has ideas for how to make things clearer or more consistent
• Likes working with Elixir (or wants to learn!)

Then head over to our GitHub repo, explore the public_api_v2 project, and jump into the issues or discussions.

We’ll also be publishing RFCs for upcoming API design decisions—keep an eye out.

What’s NextHere’s what’s coming soon:

• Better error handling across all endpoints
• More complete OpenAPI coverage
• Continued unification of all service interactions behind the gateway
• Easier ways to discover and explore available resources

We’re far from done, but we’re on a solid path—and the developer experience is already better than it was just a few months ago.

Final ThoughtsRebuilding an API isn’t glamorous work. It involves digging into edge cases, rethinking old assumptions, and making a lot of small decisions that add up to a big difference.

But the result? A cleaner, more consistent, and more enjoyable developer experience.

Check out the full talk from Semaphore Summit on YouTube if you want to dive deeper into the architecture and see the code in action.

Thanks for reading—and we hope to see you in the pull requests.

The post One API to Rule Them All: Building a Cohesive API for Semaphore appeared first on Semaphore.

View Details

AI and Large Language Models (LLMs) have entered our lives and jobs in the last few years, helping us simplify and automate tasks and processes.

For IT professionals, AI is helping streamline workloads, speed up development cycles, implement new features faster, and reduce bugs and errors. But this is not all that AI can do for us.

Managing technical debt is a task that requires time and effort from developers, architects, and managers, but it is also a task where AI can be of great help.

In this article, you will learn:

• What technical debt is.
• How AI can help you reduce technical debt.
• Abilities and frameworks that can help you manage your technical debt.
• Challenges and limitations of using AI to reduce technical debt.

Let’s dive into it!

What is Technical Debt?Technical debt refers to the cost incurred when choosing short-term solutions over long-term ones. It happens when we make quick fixes instead of investing more time and resources in improving the overall project we are working on.

There are different types of technical debt. Or, more precisely, technical debt comes with different costs, which can be classified at a high level as follows:

• Code debt: This type of technical debt occurs when you write code without considering its maintainability, readability, and scalability. Typical examples include spaghetti code, lack of modularity, hardcoded values, or insufficient use of design patterns. Many times, even if you wrote excellent code for this particular feature/problem set, the business moves on (think of a bug report which is actually a feature request) but the code did not. sigh
• Documentation debt: Let’s be honest: this is one of the most common forms of technical debt. It occurs when documentation for code, APIs, systems, or processes is missing, outdated, or incomplete, making it difficult for others to understand and contribute effectively to a project they don’t know much about. And even though it’s “easier” (less time intensive) to write docs, you’re always better off writing test cases (which at least have a chance of letting you know if/when the business has moved on but your code has not).
• Testing debt: When you skip writing tests or fail to maintain existing test suites, you create testing debt. This leads to increased risk during software releases and makes it harder to identify and fix issues promptly. Insufficient or missing automated tests lead to poor test coverage and increased risk of bugs.
• Architecture debt: This form of technical debt occurs when architectural decisions made early in the project lifecycle lead to suboptimal designs that become increasingly challenging to modify or extend over time. Common cases include monolithic systems that should be modularized, poor database design, or lack of microservices where appropriate.
• Infrastructure debt: This kind of debt arises when infrastructure components such as servers, databases, networks, or cloud services are not properly designed, maintained, or scaled to meet evolving needs. Common scenarios include manual deployment processes, lack of CI/CD pipelines, or reliance on legacy hardware.
• Security debt: Security debt is arguably the most dangerous kind of technical debt because it can lead to serious consequences if left unaddressed. It results from neglecting security best practices, inadequate threat modeling, or failing to implement proper access controls and data protection measures. This can expose sensitive information and compromise system integrity.

If you’ve ever worked on a legacy codebase or an old project, you know what technical debt means—and I know you don’t like it at all.

However, if you want to be completely honest, you have to admit that every line of code you write is potentially technical debt. You’d like that line of code to stay there as it is forever, but the reality is that it won’t happen. So why not test it now? Why not create living tests immediately? These tests not only ensure your code works as expected but also serve as the best form of documentation for future developers who will work on this codebase.

Well, the speed of development processes today is so important that we often find ourselves in situations where we need to deliver something quickly and efficiently. Sometimes, this means sacrificing some quality aspects of our codebase, unfortunately. However, this doesn’t mean that we shouldn’t try to minimize these sacrifices and keep our codebase healthy and well-maintained. The impact of technical debt can be severe in the long run, affecting companies in different areas, such as:

• Reduced development speed: Today, you’re running fast to deliver new features while accumulating technical debt because you don’t document your code and have low test coverage. Tomorrow, you’ll spend hours debugging problems caused by your previous actions and lose precious time trying to figure out how things work again. This will slow down the development of new functionalities as you’ll need to work around existing issues, such as messy code, poor architecture, or lack of automation.
• Increased maintenance costs: As mentioned before, technical debt increases the complexity of maintaining and updating the codebase. This can result in higher ongoing expenses due to longer development cycles, more frequent bug fixes, and potential rework required to address underlying issues.
• Decreased product quality: Technical debt often results in more bugs, performance issues, and system failures, which can degrade the overall quality of the product. This can also mean that users may encounter unexpected behavior or crashes, leading to negative user experiences, decreased customer satisfaction, and an increase in support tickets.
• Delayed time-to-market: The longer it takes to develop and release new features, the slower your company can respond to market demands and capitalize on opportunities. This delay can put you at a competitive disadvantage compared to competitors who invest in continuous improvement and innovation.
• Increased risk of security breaches: Poorly managed technical debt can introduce vulnerabilities and security risks into your application. If attackers exploit these weaknesses, they could gain unauthorized access to sensitive data, disrupt operations, or cause financial losses.

How AI Can Help Reduce Technical DebtThe major difficulty of managing technical debt is that it is created daily with every line of code, so it requires a continuous effort to monitor and improve the codebase. This means you should have a process that constantly reviews and refactors code, updates documentation, adds and improves tests, verifies that security has been managed to address new issues, and ensures that everything works smoothly together.

It’s hard, I know.

The good news is that AI can help you reduce technical debt. In particular, AI coding assistants can be integrated into your workflow and help you reduce technical debt by operating in the following areas:

• Code analysis and refactoring: AI-powered abilities can analyze your codebase and provide suggestions for improvements, such as identifying unused variables, optimizing algorithms, or suggesting better naming conventions. These recommendations can help you clean up your code and eliminate unnecessary technical debt.
• Automated testing: AI assistants can generate test cases based on your codebase, ensuring comprehensive coverage across various scenarios. This saves time and effort while minimizing the risk of introducing new defects through manual testing.
• Documentation generation: AI assistants can create technical documentation by extracting information from your code comments and source files. A typical example is the generation of documentation for functions, classes, or modules.
• Security assessment: Some AI abilities (like DeepCode – now part of Snyk) are specifically designed to scan your codebase for potential security vulnerabilities and suggest remediation strategies. This way, you can mitigate the risk associated with technical debt related to insecure coding practices, including detecting outdated or vulnerable dependencies.

In other words, AI can help you reduce technical debt in real time by integrating AI coding assistants into your workflows. This is revolutionary because it allows you to catch technical debt as soon as possible and prevent it from becoming a bigger problem later on.

Tools for AI-Driven Technical Debt ReductionIn a recent post, we reviewed the best AI coding assistants for 2025.

This is the chance to expand more this list to give you more options to choose your AI coding assistant to help you reduce technical debt:

• Cline: Powered by Claude 3.7 Sonnet’s agentic coding capabilities, Cline is an AI assistant designed to operate within IDEs, assisting you with software development tasks. Now able to use “dynamic thinking”, start your conversation off with the assistant in “Plan” mode and, once satisfied with the course of action, switch over to “Act”. It will then autonomously generate and edit code, execute terminal commands, and browse the web, all while requiring user permission for each action. Learn how it works here.
• Tabby: Tabby is an open-source, self-hosted AI coding assistant, compatible with major coding LLMs like CodeLlama, StarCoder, and CodeGen. Its code completion engine is designed to understand your coding context and provide real-time suggestions that are accurate and relevant. As an open-source solution, it ensures software supply chain safety, giving you peace of mind. Learn how it works here.
• Cursor: Cursor is an AI-powered coding assistant with the primary goal of streamlining the coding process and improving overall code quality. It integrates seamlessly with IDEs, provides a chat that lets you talk with an AI that sees your codebase, and can get up-to-date information from the internet. Learn how it works here.
• Codiga: Codiga is an AI-powered coding assistant and static code analysis ability designed to help developers write cleaner and safer code. One of its most appreciated features is its ability to perform real-time static code analysis, continuously checking for bugs, security vulnerabilities, and inefficiencies as you write code. Learn more about Codiga here.
• IntelliCode: IntelliCode, developed by Microsoft, is an AI-assisted code completion ability built into Visual Studio and Visual Studio Code. Its underlying LLM is trained on thousands of high-quality repositories from GitHub with high star ratings, allowing it to provide recommendations based on best practices and common patterns observed across the development community. Learn more here.

Challenges and Limitations of Using AI for Technical DebtAs you’ve learned throughout this article, AI offers significant advantages in managing technical debt. However, it doesn’t come without challenges and limitations.

Understanding these can help you make informed decisions when integrating AI into your workflows.

Accuracy and Reliability of AI ModelsFirst of all, you have to face the truth: AI models are not infallible. They can produce false positives or negatives during code analysis, flagging issues that don’t exist or missing critical problems entirely. Additionally, AI often struggles to fully understand the context of your codebase, especially in complex or domain-specific projects and this can lead to suggestions that are irrelevant or even counterproductive, requiring careful review by developers.

So, your critical skills are still invaluable. You can’t just turn off your brain and let AI do everything for you. You have to evaluate the suggestions provided by AI and decide whether they are useful or not.

Integration with Existing WorkflowsIntegrating new tools or processes into already existing workflows is not always easy, and this also applies to AI coding assistants.

So, ensuring seamless integration between AI and your (and your team’s) current abilities, processes, and team dynamics requires time and effort. Moreover, while automation can save time, it’s essential to strike a balance between relying on AI and maintaining human oversight to ensure quality and alignment with project goals so that you are still in charge of your project objectives, meaning: you don’t just blindly delegate everything to AI.

Ethical and Security ConsiderationsAI-generated code must be carefully reviewed to ensure it adheres to security best practices. While some AI coding assistants are specifically crafted with security in mind, not all of them provide a high level of security when offering suggestions. So, overlooking this step can introduce vulnerabilities into your system, potentially leading to serious consequences.

Additionally, over-reliance on AI can create a dependency that may hinder your team’s ability to solve problems independently or innovate effectively — and this is another reason why you should never rely solely on AI for decision-making.

ConclusionIn this article, you’ve learned how AI can help you reduce technical debt. You’ve seen how AI coding assistants can assist you in analyzing your codebase, generating tests, creating documentation, assessing security, and more. You’ve also explored the benefits of integrating AI into your workflows and discussed the importance of balancing automation with human oversight.

However, remember that AI is not a magic wand that solves all your problems. It requires careful consideration and implementation to achieve optimal results. In other words: your skills are still needed and highly valuable.

The post How to Use AI to Reduce Technical Debt appeared first on Semaphore.

View Details

This article is part of our build-in-public series on Semaphore Backstage, where we share how we’re designing and building our new open-source CD workflows feature, Canvas.

From YAML to Canvas: A New Way to Ship SoftwareCI/CD pipelines are essential to build and test your software. But complexity is always increasing, and once your app spans multiple services, environments, and approval steps, your deployment logic can get hard to reason about.

That’s the challenge Canvas solves.

Semaphore Canvas is a new visual way to design, track, and automate your deployments across clouds, environments, and teams. It’s a live, interactive diagram of your delivery workflow.

It builds on top of your existing Semaphore pipelines, giving you a new layer to manage:

• Complex multi-stage rollouts
• Queued versions with manual or automatic approvals
• Dynamic promotion paths
• Run and failure history per environment

Whether you’re deploying nightly RCs or promoting releases from staging to prod, Canvas helps you ship with confidence and visibility. It’s the bridge between developer automation and human decisions.

Why We’re Building a Visual Deployment DesignerDevelopers already use Semaphore to define CI/CD workflows in YAML. But when your software grows across services and environments, reasoning about your delivery logic in code alone becomes limiting. You end up with interdependent pipelines, environment-specific rules, manual approvals, rollback logic, and more.

We wanted to make this complexity visible and manageable. That’s what Canvas is for.

Canvas lets you:

• Model your deployments visually
• Chain stages together based on events
• Add manual approvals and conditions
• Track runs and queue history
• Understand which version is where

As always, you’re still setting up jobs and pipelines with the visual editor. But now, you can orchestrate them using a visual interface that mirrors how your team thinks about environments and release flows.

Want to deploy to QA, then trigger staging and production only when tests pass? Want to auto-clean test environments after three hours? Want to roll back a release candidate that failed validation in AWS but passed on GCP?

With Canvas, you can.

Managing Real-World Deployments — Queues, Approvals, and RollbacksWe designed Canvas to reflect how teams ship software — iteratively, carefully, and often across many clouds.

Let’s say you’re releasing RC versions of your app. With Canvas, you can model a testing flow like this:

• A new GitHub release triggers a deployment to a DigitalOcean test instance
• The version enters the queue and waits for your manual approval
• You test the deployment manually
• Once you’re satisfied, you approve it for Google Cloud and AWS
• If something breaks, you can reject, rerun, or rollback the queued item

Each environment becomes a stage. Each version that flows through it is a queued item. And Canvas lets you manage it all visually.

More advanced use cases are supported too:

• Selective deployments — Skip RC versions and only promote stable builds
• Parallel targets — Fan out from a single stage to deploy to multiple cloud targets
• Approval chains — Add manual gates, per-stage conditions, or fallback paths
• Timeout logic — Auto-delete test clusters three hours after creation

And Canvas keeps your audit trail clean: every run, every approval, and every rejection is tracked and timestamped.

Building Canvas in Public with React Flow and AI PrototypingWe’re not just building Canvas — we’re building it in public.

To prototype the UI, we skipped the usual wireframing tools. Instead, we paired up with AI and started coding directly with React Flow, a powerful graph visualization library for React.

We used an AI programming assistant to:

• Scaffold the project structure
• Create reusable node types (e.g. stages, event sources)
• Add interaction logic and state management
• Style nodes, edges, and queues based on state (e.g. running, failed, approved)
• Implement interactive sidebars with run history, current status, and queue items

The result? A living prototype — where we could click, test, and explore real deployment logic.

This helped us:

• Move fast without boilerplate
• Iterate on real feedback from our team
• Share progress during internal demos
• Avoid building the wrong thing early on

It also gave us confidence that the final UI would feel intuitive for developers, not like a bolt-on abstraction.

Looking AheadCanvas is already changing how we reason about deployments — and we’re just getting started.

If you want to follow our progress:

• Subscribe to the https://www.youtube.com/@SemaphoreBackstage
• Watch us build features in real time
• Share your feedback — we’d love to hear how you’d use Canvas

The post Introducing Semaphore Canvas: Visual Workflow Automation for Deployments appeared first on Semaphore.

View Details

We’re excited to announce that Semaphore now offers Ubuntu 24.04 as one of our available build images. Starting this week, you can take advantage of the latest Ubuntu LTS release on Semaphore to run your CI/CD pipelines.

What’s NewThe Ubuntu 24.04 image ships with updated versions for all the software you love, including:

• Docker 28
• Updated runtimes for all languages
• Updated SDKs for cloud providers
• Newer versions of browsers for end-to-end testing
• Extended support and the latest patches

You can find all the details on our Ubuntu 24.04 image in the docs.

What’s Changing for You New OS image: Ubuntu 24.04 is now available alongside our existing images. * Non-breaking update*: There are no breaking changes—your existing pipelines will continue to run exactly as before.

Impact on Your PipelinesSince this isn’t a disruptive change, your current workflows will not be affected. The main advantage is simply having access to the latest Ubuntu LTS version, with all its updated packages, improved performance, and extended security support.

How to Use Ubuntu 24.04To start using Ubuntu 24.04:

1. Open your project’s pipeline configuration.
2. Select a pipeline
3. Under Agent select “Linux Based Virtual Machine”
4. Select either one of the e2 or f1 machine types
5. Select ubuntu2404

Alternatively, you can change the agent on the pipeline YAML directly. For example:

version: v1.0name: CI Pipelineagent: machine: type: f1-standard-2 os_image: ubuntu2404 Commit and push the changes. Your next pipeline run will execute on Ubuntu 24.04.

F1 & E2 Machines: Rock-Solid PerformancePlease note that the Ubuntu 24.04 image is currently supported only on our high-performance e2 and f1 machines. Both machine types deliver:

• e2 machines: Cost-effective builds with great concurrency and reliability.
• f1 machines: Top-tier speed, ideal for CPU-intensive tasks and faster feedback loops.

If you haven’t tried f1 machines yet, now’s a perfect time to see what they can do for your build times!

Looking Ahead: Ubuntu 20.04 DeprecationUbuntu 20.04 reaches official EOL on 31 May 2025. We plan to deprecate our 20.04 image shortly thereafter and will publish a follow-up announcement well in advance to help you migrate. No action is needed right now—just stay tuned for that future notice.

Thank you for choosing Semaphore! We’re committed to keeping your CI/CD environment up to date and performant. If you have any questions or feedback about the Ubuntu 24.04 rollout, please reach out to support or our post your questions on our Discord server.

The post Semaphore Now Supports Ubuntu 24.04 appeared first on Semaphore.

View Details

Semaphore is a robust and efficient CI/CD platform designed to streamline software delivery. Whether you’re a solo developer or have a large team, it helps automate builds, tests, and deployments effortlessly. While it appears straightforward on the surface, its underlying architecture is a sophisticated system composed of over 30 microservices, various communication protocols, and a strong infrastructure to ensure seamless pipeline execution.

In this article, we’ll explore the core architecture of Semaphore, break down the CI/CD process from code push to execution, and examine how developers interact with the system.

To find out more, you can listen to the podcast episode above or watch the video. Enjoy!

Understanding Semaphore’s ArchitectureSemaphore balances scalability with ease of use, making it an ideal CI/CD solution for teams of any size. Its architecture is built on a distributed system that ensures high performance, fault tolerance, and automation. Let’s explore the core technologies that power Semaphore’s reliability, scalability, and efficiency.

Core TechnologiesSemaphore is powered by Elixir, which is known for its ability to handle massive concurrency and fault-tolerant systems, making it well-suited for orchestrating thousands of CI/CD jobs. We also leverage Go for performance-critical services and are gradually replacing legacy Ruby components with Elixir. The UI layer uses JavaScript & HTML for seamless developer interaction.

• Elixir: Ideal for building highly concurrent, distributed, and fault-tolerant systems that can efficiently manage thousands of jobs in parallel.
• Go: Used in performance-critical areas where efficiency is paramount.
• Ruby: A legacy component being rewritten in Elixir to align with the rest of the stack.
• JavaScript & HTML: Used for the presentation layer.

How Semaphore Services Talk to Each OtherSemaphore services need to communicate efficiently to process jobs without delay. External interactions happen over HTTP, while gRPC ensures lightning-fast internal messaging. For background tasks, RabbitMQ keeps everything running smoothly behind the scenes.

• gRPC: Provides high-performance internal communication with strict service contracts.
• Proto Files: Define service contracts before implementation to ensure smooth integration.
• RabbitMQ: Enables asynchronous messaging, allowing background tasks to run efficiently.

Scalable and Resilient InfrastructureSemaphore is built to dynamically handle any workload. Kubernetes orchestrates and scales resources as needed, PostgreSQL stores pipeline and job data efficiently, Redis improves performance with caching, and RabbitMQ ensures smooth communication between services.

These components work together to ensure that Semaphore handles workflows smoothly, balancing speed and reliability. Now, let’s dive into how Semaphore processes CI/CD jobs.

The CI/CD Workflow in SemaphoreBefore we dive into the mechanics, let’s break down Semaphore’s two main workflows:

1. CI/CD Processing – The backbone of Semaphore, ensuring every code change is built, tested, and deployed.
2. CI/CD Management – Where users interact with Semaphore to configure environments, manage projects, and control access.

How Semaphore Runs Your CI/CD WorkflowsWhen you push a new commit, Semaphore immediately takes action to ensure a smooth build, test, and deployment process:

1. Hook Processing: Semaphore detects the commit, verifies authenticity, and prepares it for execution.
2. Pipeline Processing: The Plumber Service reads the YAML configuration, organizes dependencies, and orchestrates job execution.
3. Job Execution: The Zebra Service assigns jobs to available agents, tracks execution, and ensures tasks complete on time.
4. Notifying External Systems: Semaphore updates integrations like Slack or GitHub to keep you informed.

Workflow Steps1. Hook ProcessingFirst, Hook Processing begins by listening for incoming events. The Hook Receiver accepts webhooks from Git repositories and verifies authenticity, while the Hook Processor enriches the data, converts it into an internal format, and stores it for traceability.

2. Pipeline ProcessingOnce a webhook is validated, Semaphore interprets the pipeline definition and orchestrates execution.

• Plumber Service: Defines the pipeline structure based on the YAML configuration, ensuring proper sequencing of jobs.
• Dependency Management: Ensures jobs run in the correct order and do not execute prematurely.

3. Job ExecutionNext, the job executes. The following components make this happen:

• Zebra Service: Assigns jobs to available agents, gathers necessary resources, and tracks job states.
• Agents: Execute jobs with the necessary environment and configurations.
• Timeout Management: If a job exceeds the execution time, Zebra forcefully terminates it to prevent queue blocking.

4. Notifying External SystemsOnce execution completes, Semaphore updates external systems:

• Notification Service: Sends real-time updates via Slack or webhooks.
• GitHub Notifier: Updates commit statuses in the version control system.

Additional Job Execution MethodsBeyond code pushes, developers can:

• Rerun Pipelines: Re-execute workflows without new commits.
• Start Debug Jobs: Run specific jobs in isolation for debugging.

Managing Your CI/CD Environment in SemaphoreBeyond executing jobs, Semaphore gives developers full control over their CI/CD workflows, including environment configurations, user permissions, and system integrations. Let’s dive into how you can optimize your setup.

User Interaction with SemaphoreUsers interact with Semaphore through the UI and API, with HTTP handling external communication and gRPC managing internal services.

Keeping Semaphore SecureSemaphore ensures security at every step. Keycloak handles authentication, verifying user identities before granting access. RBAC (Role-Based Access Control) enforces fine-grained permissions, ensuring only authorized users can modify configurations or trigger builds.

Wrapping UpSemaphore is designed to simplify CI/CD without sacrificing power. With automated pipelines, secure access control, and a scalable microservices architecture, it enables developers to ship high-quality software faster.

Now that you have a clearer picture of Semaphore’s inner workings, why not try it out? Whether you’re new to CI/CD or an experienced user, there’s always something to explore. As an open-source project, Semaphore welcomes contributions from the community to continuously improve and enhance its capabilities.

🚀 Ready to get started? Try Semaphore here or contribute to our open-source project and help shape the future of software delivery!

The post Unveiling the Semaphore Architecture: How It Works Under the Hood appeared first on Semaphore.

View Details

While AI-generated code and copilots have become commonplace, their role in DevOps and infrastructure remains less defined. The tooling is improving rapidly, but the risks—from non-determinism to lack of visibility—make AI adoption in production a complex, evolving journey.

In this episode of Semaphore Uncut, Patrick Debois—Generative AI and DevOps specialist —joins Darko to share his perspective on how AI intersects with DevOps, DevSecOps, and infrastructure as code. Patrick discusses everything from generative tooling to failure handling, and what makes this era of automation both exciting and risky.

A Career Built on CuriosityPatrick’s journey through the software world has always been guided by a drive to learn. He’s worn almost every hat imaginable—tester, sysadmin, mobile developer, VP of Engineering—and that broad experience led him to the insights that sparked the DevOps movement.

What set Patrick apart early in his career was his desire to work across domains, building empathy between teams and solving the friction points he observed first-hand. That perspective would later shape the DevOps Handbook, which he co-authored alongside Gene Kim, Jez Humble, and others.

Although Patrick says most of the writing credit goes to his co-authors, many of the core ideas came from his real-world experiences and the early DevOpsDays community. The second edition, with contributions from Dr. Nicole Forsgren, introduced more data and up-to-date enterprise stories, but the foundational patterns remain relevant—even if the landscape has continued evolving.

From Virtual Production to AI in DevOpsPatrick’s entry into the AI world didn’t begin with infrastructure. During the pandemic, he started exploring virtual production—automating video and media workflows using game engines like Unreal Engine. That interest in automation and digital characters led him to experiment with generative AI, initially for rendering and voice generation.

Then ChatGPT landed, and suddenly AI had a meaningful place in software development again. Patrick was hooked. He began integrating LLMs into developer workflows, helping engineers at his company use AI tools effectively, and later started advising other companies on building more robust AI-powered systems.

What keeps him engaged? “Every day, I’m learning something new—and it hasn’t stopped for a year and a half.”

Infrastructure as Code Meets Generative AIWhile AI-generated code has caught on quickly in application development, Patrick is most intrigued by its impact on infrastructure as code and DevOps workflows.

Because IaC languages like Terraform are more structured and domain-specific than general-purpose programming languages, they present an interesting target for generation. Tools can now create entire Terraform environments, visualize them, and even highlight only the parts relevant to a given change—helping reduce cognitive load.

But the bigger question is validation.

“Anyone can generate Terraform,” Patrick says, “but how do you validate it? How do you ensure it aligns with your organization’s security and compliance rules?” Patrick points to rule-based systems and emerging standards like .aiconfig files as a way to encode domain-specific requirements into the generation and review process.

Still, challenges remain. Much of the validation burden is still on the developer, and the feedback loops between code and production are often disconnected. Patrick sees a gap between developer tooling and production observability that AI could help bridge—but hasn’t yet.

From Testing to Chaos Engineering for AIPatrick draws a compelling analogy between DevOps maturity and where AI is today.

In DevOps, teams moved from simple test automation to full CI/CD pipelines, observability, resilience engineering, and chaos testing. The AI lifecycle may follow a similar arc—from basic completions to robust feedback loops that simulate failure and enable automated recovery.

“Only about 20% of the work is automation,” Patrick says. “The other 80% is preparing for failure.”

He also emphasizes the need to teach AI systems to test themselves—or at least to validate outputs using other models, digital twins, or sandboxed environments. In incident response, for instance, Patrick envisions systems that don’t just suggest solutions, but actually try multiple fixes in parallel and report back.

The Human-AI Feedback LoopOne area Patrick finds fascinating is how AI affects engineering culture and team dynamics. As AI takes on more code generation, the role of engineers shifts toward reviewing, validating, and integrating. Juniors may be able to ship code faster, but seniors are more essential than ever to evaluate quality and ensure safety.

What’s missing, Patrick says, is tooling that supports the review process itself. He points to tools like Cursor, which offer multiple ways to review code changes—including text explanations and focused visual diffs—as the future of developer experience.

“If you always see the full picture, you lose focus. AI can help us zoom in on what matters.”

The Future of Observability & AI-Native ToolingWhile DevOps has long relied on tools like Grafana and Prometheus, there’s still no AI-native operator capable of monitoring logs and dashboards in real time and proactively suggesting or testing fixes. Patrick is hopeful—but realistic.

“I can’t wait for those tools to exist,” he says. “But we’re not there yet.”

He expects progress to come from more specialized models trained on domain-specific data (like Grafana dashboards or logs), rather than general-purpose screen readers. Generative BI tools and prompt tracing are promising steps, but the field is still emerging—and often fragmented.

Final Thoughts: Why It’s Still Day OnePatrick believes we’re only at the beginning of understanding how generative AI will impact DevOps, infrastructure, and developer workflows. While the technology is advancing quickly, organizational change takes time—and toolchains still feel disjointed.

His advice? Get hands-on.

“I call it method acting. I have to use the tools myself. I need to feel the pain points firsthand.”

That hands-on mindset has guided Patrick through every major shift in software—and it’s what keeps him excited about what comes next.

Follow Patrick Debois

📺 YouTube: Jedi Forever
💼 LinkedIn: Patrick Debois
🐦 Twitter (X): @patrickdebois

The post Patrick Debois on AI & DevOps: What’s Next? appeared first on Semaphore.

View Details

Artificial intelligence continues to evolve rapidly, giving rise to AI-driven “agent” frameworks that can autonomously handle tasks and communicate with APIs in a human-like manner. Unlike typical automation scripts, AI agents leverage large language models (LLMs) to interpret, learn, and make context-based decisions. They take on repetitive chores—like analyzing new pull requests or generating release notes—so developers can focus on what truly matters.

Why AI Agents Over Traditional Automation?Traditional automation scripts are often static, following predefined if-else rules or workflows. By contrast, AI agents adapt and refine their own instructions, making them more flexible. For example, an agent configured to detect potential security risks in PRs can learn from new vulnerabilities or updated code patterns, rather than being limited to a rigid set of checks.

Common Developer Tasks to Automate Pull request reviews: Summarize diffs, catch style issues, and highlight security risks. * Release notes: Parse commit messages for automatically generated release content. * Code reviews and triaging: Suggest improvements, identify best practice violations, and group user-reported issues by root cause. * Debug log analysis*: Surface patterns in log data and proactively recommend pipeline optimizations or environment fixes.

Project Goals and Expected OutcomesSet up an AI agent that responds to triggers like new commits and integrates with Semaphore’s CI/CD pipeline to automatically summarize and review the changes.

Setting Up the Project and Choosing Your AI Agent ToolsKey Tools and FrameworksYou have several options for building an AI agent:

• AgentGPT & AutoGPT: Out-of-the-box “autonomous” GPT-driven frameworks.
• LangChain: Comprehensive toolkit for chaining LLM prompts and external APIs.
• LlamaIndex: Data-centric approach to building context/query structures on top of LLMs.
• Vercel AI SDK: Streamlined developer experience with built-in tooling for prompt orchestration, plus integration with providers like OpenAI.

Each solution has strengths: AgentGPT/AutoGPT for faster prototypes, or lower-level libraries like LangChain and the Vercel AI SDK for robust customization.

Version Control, Plugins, and Security MeasuresKeep your AI agent code in a dedicated repository or a separate folder in your main codebase. Adhere to best practices such as branching for feature work, small commits, and meaningful commit messages. This ensures that your automated tasks remain transparent and easy to maintain.

Whether you’re linking to GitHub, Slack, Jira, or other services, store credentials securely (e.g., in Semaphore secrets). Give your AI agent the minimal required permissions—like read-only on certain repos or specific Slack channels—so you don’t compromise security.

Use caution when your AI agent interacts with external services:

• Store tokens securely (e.g., environment variables in secrets).
• Review data exposures—especially in logs.
• Rotate tokens and credentials regularly.

Defining and Automating Daily Developer TasksIdentifying Tasks and Designing Task WorkflowsList out routine tasks that could benefit from LLM-powered automation:

• Code scanning (security, style checks)
• Creating draft release notes
• Debugging suggestions from logs
• Summarizing PRs, especially large ones
• Triage and labeling of new GitHub issues

Define the input and output requirements:

• Clear triggers like pushing a commit or a tag
• Well-defined outputs like AI-generated summaries, posted comments, or updated Slack messages

Define consistent error handling:

• Retry with exponential back-off
• Notify a Slack channel or email list for manual intervention

Implementation DetailsCode Structure:

• Use a logical folder structure.
• This makes it easier to add or refine tasks without confusion.

Logging and Data Storage:

• Agents often produce valuable metadata such as logs of LLM calls, error messages, or final decisions.
• Decide on a storage approach (like local JSON logs or a time-series database) that your team can later review or analyze for improvements.

Code Reviews and Commit SummariesIn this step, we’ll explore a small Node.js agent powered by the Vercel AI SDK that automatically reviews commits, summarizes code changes, and flags security or style issues. The goal is to reduce manual overhead and let developers focus on high-impact work rather than repetitive, preliminary checks.

Benefits and Key Objectives for Commit Agent Automated Commit Triggers + Integrate your AI agent with GitHub’s webhooks or rely on Semaphore’s pipeline triggers whenever a new commit is pushed. + This ensures immediate detection of code changes and helps your team fix problems at the earliest possible time. * Intelligent Summaries & Security Checks + By leveraging GPT-4 or other LLMs, your AI agent can parse diffs and highlight potential vulnerabilities, style problems, or major architectural changes. + Reviewing a large commit becomes easier when a summarized outline is readily available. * Actionable Feedback & Suggestions + Developers can receive direct improvement tips—from refactoring opportunities to adopting best practices for dependency management. + These suggestions reduce the mental overhead required during code review. * Seamless CI/CD Integration* + With the help of Semaphore blocks and jobs, you can automatically run the code review agent removing the need for manual triggers. + Comments or logs can be posted to PR threads enabling quick, iterative feedback loops.

Getting the ProjectTo access the code review agent, clone the repository:

git clone git@github.com:ajcwebdev/semaphore-ai-agents.gitcd semaphore-ai-agents Project DependenciesThe project uses the following key dependencies:

• ai and @ai-sdk/openai – The core Vercel AI SDK and OpenAI provider packages
• zod – Helps define schemas for validating or structuring data

Environment SetupThe project requires a .env file with the following variables:

OPENAI_API_KEY=YOUR_OPENAI_KEYGITHUB_TOKEN=YOUR_GITHUB_TOKENGITHUB_OWNER=YOUR_GITHUB_USERNAMEGITHUB_REPO=YOUR_GITHUB_REPO For example, if I want my AI Assistant to review this codebase ajcwebdev/semaphore-ai-agents, I’d set GITHUB_OWNER=ajcwebdev& GITHUB_REPO=semaphore-ai-agents. If you’ve forked the repo, you will have the same GITHUB_REPO name but your own GITHUB_OWNER account name.

You’ll need to create a fine-grained personal access token on GitHub at github.com/settings/personal-access-tokens. The only permission required is read-only access to the public GitHub repository that is connecting to Semaphore.

Understanding the Code StructureThe overall project structure looks like this:

semaphore-ai-agents/├── .env├── .gitignore├── package.json├── src/│ ├── config.ts│ ├── index.ts│ ├── ai/│ │ └── review.ts│ ├── commits/│ │ └── parser.ts│ └── github/│ └── tools.ts└── .semaphore/ └── semaphore.yml Let’s examine the key components of the commit analysis agent:

1. Configuration Module (src/config.ts)This module manages environment variables and validation. The code loads environment variables and provides a validation function to ensure all required configuration is present before proceeding.

2. GitHub API Tools (src/github/tools.ts)This module handles GitHub API interactions.

This module provides two key functions:

• fetchCommitDiff – An AI SDK tool that retrieves the diff for a specific commit SHA

• fetchCompareCommits – A function that fetches all commits between two SHAs using GitHub’s compare API

• Commit Parser (src/commits/parser.ts)This module handles parsing commit ranges. The code intelligently resolves the commit ranges provided by Semaphore CI environment variables. It handles both:

• Single commits (when only SEMAPHORE_GIT_SHA is available)

• Commit ranges using GitHub’s compare API (when SEMAPHORE_GIT_COMMIT_RANGE is available)

It also converts two-dot notation (..) to three-dot notation (...) for compatibility with GitHub’s API.

1. AI Review Generator (src/ai/review.ts)This module generates the AI-powered code reviews. The module uses the Vercel AI SDK to generate intelligent code reviews. It:

2. Uses GPT-4o to analyze commit diffs

3. Provides specific instructions to summarize changes, flag issues, and suggest improvements
4. Uses the fetchCommitDiff tool to retrieve the code changes for analysis

5. Limits execution to 2 steps to optimize performance

6. Main Entry Point (src/index.ts)This is the main orchestration file that brings everything together. The main file coordinates the entire review process:

7. Validates the environment configuration

8. Resolves the commits to review
9. Generates AI reviews for each commit
10. Outputs the reviews to the console
11. Handles errors properly to ensure CI failures are reported

Semaphore CI/CD IntegrationOpen the Settings page on your Semaphore account and select the “Secrets” tab to create pr-review-secrets. Include the same environment variables inside your project’s .env file.

Semaphore Configuration FileThe project includes a Semaphore configuration file to trigger the agent on new commits. The AI output is captured in a file called ai-review.log and stored as an artifact for easy access in the Semaphore UI.

```

.semaphore/semaphore.ymlversion: v1.0name: Code Review Pipelineagent: machine: type: e1-standard-2 os_image: ubuntu2004global_job_config: prologue: commands: - checkout --depth=full - npm installblocks: - name: Commit Review task: secrets: # Create pr-review-secrets in the Semaphore UI or with the Semaphore CLI - name: pr-review-secrets jobs: - name: Review Commits commands: # Run AI review script, capturing the output in a log file - npx tsx src/index.ts | tee ai-review.log # Push AI review log as a job artifact, view from Semaphore UI Artifacts tab. - artifact push job ai-review.log dependencies: []

``` This configuration:

• Sets up a build pipeline that runs on Ubuntu 20.04
• Checks out the full commit history to ensure proper diff generation
• Installs dependencies and runs the review script
• Saves the AI review output to ai-review.log, then pushes it as a job artifact
• Uses a Semaphore secret named pr-review-secrets to securely provide the required environment variables

When Semaphore triggers a build on new commits, it will set SEMAPHORE_GIT_COMMIT_RANGE (or SEMAPHORE_GIT_SHA if only one commit is pushed). The script retrieves each commit’s diff and generates an AI review.

The review is stored both in the Semaphore logs (via tee) and as an artifact (ai-review.log). You can open the job log to read the review directly:

Or go to the Artifacts tab in your Semaphore job page to download the file:

Benefits of This Modular ApproachThe repository’s modular structure offers several advantages:

1. Separation of concerns: Each file handles a specific aspect of the application
2. Testability: Components can be tested in isolation
3. Maintainability: Smaller focused files are easier to understand and modify
4. Reusability: The modules can be reused across different parts of the application

By leveraging this AI-powered commit review agent, your team can gain immediate insights about each commit in a push or pull request, helping to identify potential issues before they reach human reviewers. Storing the artifact with the AI’s output makes it easy for the whole team to access and reference code changes.

ConclusionRecapWe started by discussing the motivations behind AI agents and why they can surpass traditional, rule-based automation. We walked through the basics of setting up your project, selecting the right frameworks, and defining your automated workflows. Then we dove into how to build an AI agent for commit analysis that provides summaries, security checks, and style feedback.

Best Practices Prompt Engineering: Craft system messages and prompts that guide the LLM to produce actionable feedback. * Secure Credential Management: Always store tokens and secrets in a protected environment, such as Semaphore’s secrets. * Monitoring & Iteration: AI agents improve with continuous prompts, data logs, and examples. Regularly review their suggestions for accuracy. * Limiting Permissions*: Provide the minimal necessary access—reduce the risk of unintentional modifications.

Next Steps & Future Improvements1. Advanced Analytics: Integrate more robust usage metrics and dashboards to refine the agent’s suggestions over time. 2. Complex Integrations: Explore adding JIRA or Asana tasks automatically based on your agent’s analysis or code updates. 3. Refined Prompting Strategies: Use conversation memory or chain-of-thought techniques for even more accurate agent responses. 4. Support for Additional Developer Tools: Incorporate tools like ESLint, SonarQube, or custom security scanners for deeper code insights.

Call to ActionClone or fork the example repository, add your own environment variables, and integrate the AI agent with your existing CI/CD pipeline on Semaphore. You’ll quickly see how these agents can reduce your overhead while improving code quality and release workflows.

Resources* Semaphore Documentation * Vercel AI SDK and OpenAI Provider * LangChain * GitHub Actions for LLM-based PR review * AgentGPT and AutoGPT

The post How to Build an AI Agent to Help with Daily Tasks appeared first on Semaphore.

View Details

Ever hit ‘run’ on your CI/CD pipeline, only to grab a coffee… and it’s still not done when you get back? We’ve all been there. Slow builds can feel like a productivity killer — but there’s a better way.

Semaphore’s self-hosted agents give you control — delivering faster builds, improved scalability, and enhanced security by running agents within your infrastructure. Whether you’re scaling build pipelines for a growing team or need flexible deployment options, self-hosted agents can solve your CI/CD headaches.

Let’s dive into how they work — and how you can get the most out of them.

To learn more about self hosted agents, you can listen the podcast or watch the video below:

What Are Semaphore’s Self-Hosted Agents?Semaphore’s architecture is built for performance and flexibility, consisting of two key parts — the brains and the brawn:

• Control Plane: The brains behind Semaphore that orchestrates pipelines, manages secrets, and controls job execution.
• Agent Pool: The brawn — a set of workers that execute your build and deployment jobs. These can run on Semaphore’s infrastructure or as self-hosted agents within your environment.

How to Register a Self-Hosted Agent in Semaphore (Step-by-Step Guide)When you spin up a self-hosted agent, the first thing it does is knock on the control plane’s door, asking for permission to join the party.

Here’s what happens next:

• The agent sends a registration request to the control plane. All agents of the same type use the same registration token.
• The control plane assigns the agent a unique access token for secure communication.

Semaphore enforces quotas to ensure your infrastructure doesn’t accidentally spawn an army of rogue agents. Each organization has a maximum number of agents allowed to run at once. To ensure this is managed properly across multiple self-hosted hub instances, Semaphore uses Postgres advisory locks to guarantee consistency — no unexpected agent overloads.

Mastering the Semaphore Agent Lifecycle for Reliable BuildsOnce registered, an agent moves through predictable states.

• Waiting for Job: The agent stays idle until assigned a job.
• Running Job: The agent executes the assigned job.
• Finished Job: The agent either disconnects or returns to the waiting state based on your configuration.

Semaphore’s sync protocol efficiently manages state transitions, ensuring agents respond quickly while avoiding excessive system load.

How Semaphore Controls CI/CD Logs for Maximum EfficiencyLogs are a lifeline when debugging. Semaphore’s Log Hub service makes sure they stay organized and efficient:

• Receives log batches from agents.
• Stores active logs in Redis for fast access.
• Archives completed job logs in object storage (e.g., S3) for long-term retention.

But what if your build spits out a massive wall of logs? Semaphore caps log uploads to prevent system overload. Don’t worry — the agent will still keep those logs locally, so you can upload them later as artifacts for detailed debugging.

Scaling Semaphore Agents for Fast, Reliable DeploymentsScaling agents can be tricky — unless you let Semaphore do the heavy lifting. Whether you’re running AWS, Kubernetes, or your own setup, Semaphore’s tools help you scale smart, without extra work:

• On AWS: Tools like AWS CDK automate the deployment of auto-scaling groups, manage AMIs for consistent environments, and configure IAM roles securely.
• On Kubernetes: Custom controllers manage agent scaling dynamically.

Semaphore’s dedicated tools simplify deployment and ensure your agents scale as your workloads grow.

Key Takeaways for Developers Faster Builds, Less Waiting: Self-hosted agents eliminate bottlenecks, improving your CI/CD performance. * Flexible Deployment Options: Deploy agents on AWS, Kubernetes, or your own infrastructure to suit your needs. * Reliable Scaling:* Semaphore’s smart architecture handles dynamic scaling without manual intervention.

Try Semaphore’s Self-Hosted Agents TodaySemaphore continues to evolve with support from the open-source community, making it easier than ever to build and deploy quality software at scale.

🚀 Ready to stop waiting on slow builds and ship faster? Get Started with Semaphore’s Self-Hosted Agents Now.

Have questions or insights on self-hosted agents? Join the conversation in our community!

The post How to Scale CI/CD With Self-Hosted Agents appeared first on Semaphore.

View Details

Acceptance testing is essential for verifying that software meets user requirements by ensuring it functions as expected. It validates the system’s behavior against predefined criteria, providing confidence in the final product. However, traditional acceptance testing methods often struggle with communication gaps between technical teams and non-technical stakeholders, leading to costly rework and project delays.

A common scenario highlights this issue: a business analyst (BA) gathers requirements from a client and then communicates them to the development team. Despite best efforts, different interpretations can cause discrepancies between what the client envisions and what the developer delivers. These misunderstandings in requirements are a major challenge for successful software delivery.

By improving collaboration through methods like Behavior-Driven Development (BDD), teams can bridge the gap between stakeholders, minimize misunderstandings, and ensure alignment from the start.

Behavior-Driven Development (BDD) is a software development methodology that bridges the gap between business requirements and technical implementation. It encourages collaboration among developers, testers, and business stakeholders. They define together how the system should behave in a shared language.

In this article, we’ll explore how to combine BDD with acceptance testing. You’ll see how BDD transforms acceptance tests into living documentation. We’ll also discuss how to automate these tests using Semaphore CI. By the end of this article, you’ll understand why BDD is the best approach for creating effective and maintainable acceptance tests.

What is Acceptance Testing?Acceptance testing ensures a system meets user requirements and behaves as expected in real-world scenarios. It confirms the software delivers the functionality stakeholders agreed upon. It is a crucial step and can either be the final phase or come just before end-to-end testing.

Traditionally, acceptance testing was performed manually. Business analysts or QA teams would write test cases based on requirements and execute them step-by-step to confirm expected behavior. While manual testing works for small projects, it becomes time-consuming, error-prone, and unsustainable for modern software development.

Today acceptance testing relies on automation to overcome these challenges. By automating acceptance tests, teams can validate behavior continuously throughout the development lifecycle. This not only reduces manual effort but also ensures that changes do not break existing functionality.

Here’s an example of an acceptance test:

A requirement states, “The system must allow a user to reset their password.” A simple acceptance test would verify this behavior:

1. The user navigates to the “Forgot Password” page.
2. The user submits their email.
3. The system sends a reset link to the user’s email.
4. The user resets the password successfully.

The goal of acceptance testing is to confirm that the system satisfies these steps from the user’s perspective.

However, manually defining and maintaining such test scenarios often leads to inconsistencies and misunderstandings. This is where BDD comes into play. By structuring acceptance tests as executable specifications, BDD eliminates ambiguity and improves team collaboration.

In the next section, we’ll explore how BDD works and why it’s a good idea to combine it with acceptance testing.

How BDD WorksNow, let’s dive deeper into how BDD works and why it’s transformative for acceptance testing.

Structure of BDDThe core idea of BDD is to write scenarios that define how the system should behave. These scenarios use natural, human-readable language that both technical and non-technical team members can understand. A fundamental aspect of BDD is the Gherkin syntax, which structures acceptance criteria into a readable format. It uses the Given-When-Then pattern. The tests express the user’s point of view.

• Given: Describes the preconditions or context.
• When: Specifies the action or event.
• Then: States the expected outcome.

For example, let’s revisit the password reset requirement:

Feature: Password Reset Scenario: Successful password reset request Given the user is on the "Forgot Password" page When the user submits their valid email address "user@example.com" Then the system sends a password reset link to "user@example.com" You can also enhance the clarity and expressiveness of scenarios by using additional keywords like And, But, and Examples.

• And: This keyword allows you to chain multiple actions or expectations in a natural way. For example, “Given the user is logged in, and the user has admin privileges.”
• But: This is a way to introduce an exception or a contrasting condition. For example, “When the user clicks the button, but the internet connection is lost.”
• Examples: The Examples keyword is useful for data-driven testing. It allows you to run the same scenario with multiple data. For example, “Given I am on the login page when I enter the following username and password: Examples Username | Password | Expected Result.”

This is how BDD transforms business requirements into executable scenarios that serve as documentation and tests.

The Benefits of BDDBDD brings several advantages to software development:

• Shared Understanding: Scenarios are written in plain language, enabling collaboration between stakeholders, developers, and testers.
• Executable Specifications: Scenarios are not just text—they can be automated and executed to validate behavior.
• Reduced misunderstandings: By involving stakeholders in defining behaviors, BDD minimizes miscommunication and ensures clarity in requirements.
• Lower costs: Preventing misunderstandings and errors early in the development cycle saves time and resources.
• Safer releases: Scenarios act as living documentation, ensuring new changes don’t disrupt existing functionality.

When to Use BDD?BDD scenarios are not necessarily mandatory for every product specification. They are most useful for complex specifications where there is a risk of misunderstanding or when more thorough testing is needed. For simple tasks, like color or text changes, using BDD scenarios would be inefficient and unnecessary.

BDD and User StoriesIn practice, teams can effectively integrate BDD with user stories in Jira. For each user story, they create a corresponding BDD scenario, often in the Given-When-Then format, which directly translates the acceptance criteria into automated tests.

Jira User Story:

As a user, I want to be able to reset my password using my email address so that I can regain access to my account if I forget my password.

Acceptance Criteria (in the user story):

The user should be able to request a password reset by entering their registered email. The user should receive a password reset email with a link to reset their password. The link should expire after 30 minutes.

Corresponding BDD Scenario:

Feature: Password ResetScenario: User requests a password resetGiven the user is on the login pageWhen the user enters their registered email and clicks "Reset Password"Then the user should receive a password reset emailAnd the email should contain a link to reset the passwordAnd the reset link should expire in 30 minutes BDD ChallengesWhile BDD offers significant benefits, teams often face challenges when implementing it effectively.

1. Maintaining Tests
2. Challenge: Keeping tests updated as the system evolves.
3. Why it Happens: Poorly written tests may become outdated or too complex.
4. Solution:
   • Balance detail and simplicity.
   • Regularly review and refactor tests.
   • Focus on aligning tests with evolving requirements.
5. Team Collaboration
6. Challenge: BDD relies heavily on collaboration between developers, testers, and stakeholders.
7. Why it Matters : Without strong teamwork, tests may not accurately reflect the system’s intended behavior.
8. Solution:
   • Foster open communication in Agile environments.
   • Build cross-functional teams with shared goals and understanding.
9. Adopting the BDD Mindset
10. Challenge: Shifting to BDD can feel daunting, especially for teams used to traditional testing methods.
11. Why it Happens: BDD has a steep learning curve, leading to resistance.
12. Solution:
    • Emphasize benefits like improved collaboration and alignment with business goals.
    • Share success stories and celebrate small wins.
13. BDD Anti-Patterns
14. Feature-Coupled Step Definitions: These occur when step definitions are tightly coupled to specific scenarios or features, making them hard to reuse. For example:

Given the user is logged inWhen the user clicks on "Add to Cart"Then the item is added to their shopping cart If this step relates to a single feature like “Add to Cart,” it might not be reusable for similar actions elsewhere, such as “Add to Wishlist.”

• Solution:
  • Use generic, reusable steps:

When the user performs an action on "Add to Cart"Then the system updates the corresponding data * Conjunction Steps: These steps combine multiple actions into one, making tests harder to understand and debug. For instance:

When the user fills out the form and clicks submit It’s hard to read and unclear whether the issue lies with filling out the form or with clicking submit.

Solution: Break into smaller, clear steps:

When the user fills out the formAnd the user clicks submit 5. Trivial Functionality Tests: These are scenarios that test functionality already covered by unit tests, adding unnecessary complexity. * Why it Happens: Some teams might feel pressure to achieve high test coverage and include trivial cases in scenarios. Others may treat BDD as a replacement for all types of testing. * Solution: + Keep scenarios focused on what the system does, not how it works internally. + Use BDD to describe high-level behaviors or system-level interactions that matter to stakeholders.

Popular BDD ToolsSeveral tools support BDD by allowing teams to write and execute tests based on behavioral specifications:

• Cucumber: Supports various programming languages and offers strong Gherkin support, making it highly popular. It can even sync with Jira to automatically link the scenarios to the relevant user stories.
• Behave: A Python-based BDD framework ideal for teams working with Python projects.
• Reqnroll: A popular BDD framework for .NET developers.

The Links Between BDD and Acceptance TestingExecutable Specifications

BDD turns acceptance tests into executable specifications, aligning technical implementation with business goals. Using Gherkin, teams write testable scenarios that serve as a single source of truth for all stakeholders. Tip: Structure scenarios with the Given-When-Then format for clarity and consistency.

Behavior as Acceptance Criteria

BDD emphasizes translating vague requirements into clear, testable scenarios. For example:

| Traditional Requirements | BDD Acceptance Criteria | | --- | --- | | “Allow password reset” | Scenario: Password Reset Request |

Collaboration Across Teams

BDD fosters a shared understanding between developers, testers, and stakeholders. Its simple Given/When/Then syntax allows everyone to contribute to defining behaviors.

Linking BDD to acceptance testing enhances clarity, collaboration, and confidence in meeting customer expectations.

Automating BDD Acceptance Tests with CI/CDIntegrating BDD tests into your CI/CD pipeline ensures continuous validation of features. Automated tests catch regressions early, meeting user expectations before deployment.

Here’s a simplified CI/CD pipeline flow when using BDD-based acceptance tests:

With tools like Semaphore CI, you can configure this workflow to run on every commit, branch, or merge request.

Automating Acceptance Tests with Semaphore CIIn this example, we’ll write a BDD test for a Python project using the Behave framework. We’ll automate the BDD tests using Semaphore CI’s pipeline.

First, create a virtual environment to safely install Python packages:

python3 -m venv myenvsource myenv/bin/activate Install Behave:

pip3 install behave Create a file named password_reset.feature in a features directory. This file contains the BDD scenarios using Gherkin syntax.

Feature: Password Reset In order to regain access to my account As a registered user I want to be able to reset my password via email Scenario: User submits a valid email for password reset Given the user is on the "Forgot Password" page When the user submits a valid email "user@example.com" Then the system sends a password reset link to "user@example.com" Scenario: User submits an invalid email for password reset Given the user is on the "Forgot Password" page When the user submits an invalid email "invalid_email" Then the system shows an error message "Please enter a valid email address" Create a file named password_reset_system.py to implement the system functionality:

class PasswordResetSystem: def __init__(self): self.valid_emails = [] # Store valid emails for testing purposes def add_valid_email(self, email): """Method to add valid email addresses.""" if self.is_valid_email(email): self.valid_emails.append(email) else: raise ValueError(f"Invalid email format: {email}") def reset_password(self, email): """Simulate the password reset request.""" if self.is_valid_email(email): return f"Password reset link sent to {email}" else: return "Please enter a valid email address" def is_valid_email(self, email): """Helper method to check if email is in valid format.""" return "@" in email and "." in email # Very basic email validation (see https://pypi.org/project/email-validator/) Create a main.py file where you initialize the PasswordResetSystem class and interact with it.

from password_reset_system import PasswordResetSystemdef main(): # Create an instance of the PasswordResetSystem system = PasswordResetSystem() # Example of adding a valid email try: system.add_valid_email("user@example.com") print("Valid email added!") except ValueError as e: print(e) # Simulate a password reset request with a valid email response = system.reset_password("user@example.com") print(response) # Output: Password reset link sent to user@example.com # Simulate a password reset request with an invalid email response = system.reset_password("invalid_email") print(response) # Output: Please enter a valid email addressif __name__ == "__main__": main() When you run main.py, you will see output similar to this:

$ python3 main.py Valid email added!Password reset link sent to user@example.comPlease enter a valid email address Now, create a steps.py file in a steps directory to define the steps in your scenarios. Behave will look for step definitions that match the steps in the feature file.

from behave import given, when, thenfrom password_reset_system import PasswordResetSystem# Initialize the system instancesystem = PasswordResetSystem()@given('the user is on the "Forgot Password" page')def step_given_user_on_forgot_password_page(context): context.page = "Forgot Password" context.system = system # Assign the system object to context for use in other steps@when('the user submits a valid email "{email}"')def step_when_user_submits_valid_email(context, email): context.response = context.system.reset_password(email)@when('the user submits an invalid email "{email}"')def step_when_user_submits_invalid_email(context, email): context.response = context.system.reset_password(email)@then('the system sends a password reset link to "{email}"')def step_then_system_sends_reset_link(context, email): assert context.response == f"Password reset link sent to {email}", f"Expected reset link for {email}, got {context.response}"@then('the system shows an error message "{message}"')def step_then_system_shows_error_message(context, message): assert context.response == message, f"Expected error message '{message}', got {context.response}" Key points:

• This file defines the Python functions that implement the actions from the Gherkin steps. For example, when the user submits a valid or invalid email, the PasswordResetSystem class simulates the system’s behavior and returns the appropriate response.
• We use assert to perform assertions. For example, we assert that the response from the reset_password method matches the expected output.

To run the tests, execute this command:

$ python -m behave You should see output similar to this:

$ python -m behaveFeature: Password Reset # features/password_reset.feature:1 In order to regain access to my account As a registered user I want to be able to reset my password via email Scenario: User submits a valid email for password reset # features/password_reset.feature:7 Given the user is on the "Forgot Password" page # steps/steps.py:8 0.000s When the user submits a valid email "user@example.com" # steps/steps.py:14 0.000s Then the system sends a password reset link to "user@example.com" # steps/steps.py:22 0.000s Scenario: User submits an invalid email for password reset # features/password_reset.feature:12 Given the user is on the "Forgot Password" page # steps/steps.py:8 0.000s When the user submits an invalid email "invalid_email" # steps/steps.py:18 0.000s Then the system shows an error message "Please enter a valid email address" # steps/steps.py:26 0.000s1 feature passed, 0 failed, 0 skipped2 scenarios passed, 0 failed, 0 skipped6 steps passed, 0 failed, 0 skipped, 0 undefinedTook 0m0.000s Generate a requirements.txt file, which includes all the Python dependencies you’ve installed in your environment:

$ pip freeze > requirements.txt Run the deactivate command to deactivate the virtual environment when you’re done.

Let’s move on to the CI part.

Semaphore expects a .semaphore folder at the root of your project, as well as a semaphore.yml file containing the CI configuration for the project:

version: v1.0name: Pythonagent: machine: type: e2-standard-2 os_image: ubuntu2204blocks: - name: "Install Dependencies" task: prologue: commands: - sudo apt-get update && sudo apt-get install -y python3-dev jobs: - name: pip commands: - checkout - cache restore - pip download --cache-dir .pip_cache -r requirements.txt - cache store - name: "Run Tests" task: prologue: commands: - checkout - cache restore - pip install -r requirements.txt --cache-dir .pip_cache jobs: - name: Run BDD Tests commands: - python -m behave Key points:

• The e2-standard-2 machine type has 2 CPUs and 4GB of RAM, suitable for standard workloads.
• The commands in the prologue execute before the block’s jobs run.
• The “Install dependencies” block updates the system and installs Python dev tools. It also caches Python dependencies to speed up future runs.
• The “Run Tests” block restores dependencies from the cache and installs them. Then it runs the BDD tests using python -m behave.

To execute the tests in the pipeline, you need a Semaphore account. Visit the signup page and choose either GitHub or Bitbucket for registration. In this example, I’ll use GitHub.

Next, create a new project by clicking the “+ Create new” button.

Add a GitHub repository. For example, I’ll use the demo project I created for this tutorial.

Semaphore will automatically initialize the project. As you already have the semaphore.yml file in place, you can use it directly in the pipeline workflow. Each push to the project triggers the pipeline.

You can expect results like this:

The “Run BDD Tests” job displays the test results. You should see the same output as you did earlier in your Terminal:

You can find the complete source code for this demo in my GitHub repository.

ConclusionIn this article, we examined BDD’s functionality, benefits, challenges, and its integration with acceptance testing to enhance collaboration and develop effective testing strategies. Utilizing BDD with acceptance testing boosts team confidence by ensuring features address user needs.

You also learned how to implement a BDD test and automate it within the Semaphore CI pipeline.

Start small—identify critical business scenarios, write them in the Given-When-Then format, and automate them as acceptance tests. Once you integrate them into your CI/CD pipeline, these tests will ensure quick feedback and more reliable software delivery.

The post Accelerate your CI/CD Pipeline with BDD and Acceptance Testing appeared first on Semaphore.

View Details

I’m thrilled to share transformative news about Semaphore’s evolution. After serving developers through our commercial offering for over 12 years, we’re taking a significant step by open sourcing our core platform under the Apache 2.0 license. This decision reflects our belief that the most powerful solutions emerge when developers can deeply understand, modify, and enhance their tools.

Why the changeWhen we started in 2012, commercial, closed-source platforms like GitHub and Heroku showed the way forward in cloud services. Times have changed dramatically since then: Docker and Kubernetes have standardized cloud and on-prem deployment, security has become crucial for all companies, and organizations like Gitlab and HashiCorp have proven sustainable open source business models.

Yet today’s CI/CD landscape presents developers with an uncomfortable choice: they can either use open source tools that offer flexibility but struggle with scalability and user experience, or accept vendor lock-in from platforms where CI/CD is just another checkbox feature. Neither path fully serves modern development teams’ needs.

Semaphore has grown beyond our initial vision, shaped by thousands of hours of collaboration with our amazing customers. Each feature and improvement in developer experience has emerged from deep partnerships with teams solving real-world delivery challenges. Opening our entire platform to developers worldwide is the natural next step – an opportunity to expand this collaborative spirit to a broader community.

The new modelOn February 26th, the core platform that powers Semaphore will be available on GitHub under Apache 2.0 license. This production-grade codebase, from our pipeline orchestration engine to our workflow builder, processes millions of workflows across thousands of organizations today.

Semaphore will be available in three editions to meet different needs:

1. Semaphore Cloud remains our fully-managed service, offering both pure cloud and hybrid solutions for teams that want a seamless experience and the fastest CI/CD pipelines on the market, and access to all advanced features like pipeline promotions and flaky test detection.
2. Semaphore Enterprise Edition provides on-premises deployment with advanced security and compliance features that larger organizations require.
3. Semaphore Community Edition represents our core platform, now open source and free to use on your own infrastructure. This gives developers complete control to run and customize Semaphore for their specific needs.

Each edition maintains clear boundaries between open source and commercial features, ensuring a sustainable model where community innovation and enterprise reliability reinforce each other.

For our current usersWe have carefully designed this transition to ensure continuity for all our users. If you’re using Semaphore Cloud today, your experience stays the same – you’ll continue to enjoy our fully-managed service and industry-leading pipeline performance. Enterprise On-prem customers maintain access to all their current features, with the added benefit of being able to examine and extend the core platform code.

The open sourcing of our core platform creates new opportunities for all our users. You’ll be able to better understand how Semaphore works, contribute improvements that matter to your team, and build custom integrations with unprecedented flexibility.

Our commitment to security and reliability remains unchanged – the same components that have reliably processed millions of your workflows are the ones we’re opening to the community.

For developersThis isn’t another half-baked open source release. We’ve spent a decade building a developer-focused CI/CD platform that makes complex workflows clear and simple, at any scale. Semaphore’s code and infrastructure have passed a comprehensive security audit by Cure53, and we’re opening production-grade code that processes millions of jobs daily.

Our mission now is to build the next major milestones together with the community. We’re embracing full transparency by moving all our product and engineering discussions to public channels on Discord, GitHub, and YouTube. We believe this “building in public” approach, combined with our proven UX and scalability, creates the foundation for a new standard in CI/CD that developers have been waiting for.

The path forwardOver the next three weeks, we will be preparing for the open source release by:

• Transitioning to our new home at semaphore.io
• Publishing comprehensive documentation, including architecture diagrams and contribution guides
• Preparing the main repository with issue templates and CI/CD pipelines

The journey culminates with Semaphore Summit, our online community event starting February 26. The Summit opens with a keynote announcing Semaphore’s open source release, where we’ll make the code publicly available and share our vision for the future of CI/CD.

Later that day, I’ll host a fireside chat to dive deeper into our technical roadmap and answer your questions about contributing to Semaphore.

The Summit continues over three days, with a few focused sessions each day to make it easy to attend alongside your regular work. You’ll hear from teams who’ve built remarkable delivery pipelines, join technical deep-dives into advanced patterns, and participate in hands-on contribution workshops. We’ve designed each day’s schedule to be both valuable and manageable, whether you join us for specific sessions or the entire event.

Get involvedJoin us at Semaphore Summit:

• February 26-28, 2025
• Online and free
• Register here

This marks the beginning of a new chapter in CI/CD – one where developer experience, enterprise reliability, and community innovation come together. With production-grade foundations and a commitment to building in public, we’re creating the open platform we wish had existed when we started our journey. I’m excited to see what we’ll build together.

Darko,

Semaphore CTO and co-founder

The post Semaphore is Going Open Source in 30 days appeared first on Semaphore.

View Details

As we step into 2025, the tech landscape is set for another transformative year. From artificial intelligence (AI) making strides in automation and creativity to the looming potential of quantum computing, this year promises to be pivotal. Let’s explore the key developments and expected tech trends 2025.

AI: Growing Influence and ChallengesLegal Battles Over AI Training Data
AI continues to make headlines, but not without controversy. OpenAI recently won a round of lawsuits from news outlets over its use of trending data for training AI models. However, this doesn’t mean the fight is over—lawsuits like the one from The New York Times, alleging unauthorized use of journal articles, could set important precedents for the future of AI training. If these cases swing in favor of content creators, it could change how AI companies operate, particularly in how they source their data.

AI-Generated Code on the Rise
AI-powered tools are increasingly being used to write code. Google reported that 20% of code written in 2024 was AI-generated—a figure that’s expected to grow. Developers are divided on this, though. While many see AI as a helpful tool, around 40% of developers don’t fully trust the quality of AI-generated code. Concerns about technical debt and reliability may push companies to demand better-performing models this year.

Tech trends for 2025: AI Agents Gaining Momentum
AI agents—software designed to perform tasks on your behalf—are expected to be a big focus in 2025. Companies like Anthropic, OpenAI, Google, and Microsoft are all diving into this space. These agents can handle tasks ranging from personal assistance, like ordering food or booking trips, to business processes, such as automating workflows or writing code. While this technology is exciting, its adoption will depend heavily on how well it can be integrated into daily life.

Small AI Models and Edge Computing
Not all AI models are massive and resource-hungry. In 2025, smaller, more efficient AI models that can run directly on devices like phones or laptops will continue to emerge. These models make AI features feel more seamless and accessible, especially for specialized tasks. Open-source projects are leading this tech trend for 2025, allowing users to fine-tune models for their specific needs without relying heavily on cloud-based systems.

Tech Trends for 2025: The Rise of AI in Video and Creative Industries
AI’s ability to generate content is expanding from images to videos. OpenAI’s Sora and Google’s Veo2 models are already creating 20-second video clips from text prompts. While this technology is still in its early stages, it could disrupt industries like stock video, much like how AI reshaped stock photography. The tech trends for 2025 are likely to see even more advancements in text-to-video capabilities, making high-quality video production faster and more accessible.

Security and AI-Driven Scams
AI’s potential isn’t all positive. Its growing power has also led to more sophisticated scams and cyber threats. In one alarming case last year, a scammer used AI-generated deepfake video to impersonate a company executive and steal $25 million. As these technologies improve, spotting fake communications will become increasingly difficult. Businesses and individuals will need to be more vigilant to protect themselves from these advanced threats.

Tech trends for 2025: Sustainability Concerns and AI’s Energy Demands
The rapid growth of AI as a tech trend for 2025 comes with significant energy requirements. It’s estimated that by 2027, AI’s energy consumption could quadruple compared to 2024 levels. Big tech companies are already taking action to address this. For example, Microsoft is reactivating the Three Mile Island nuclear facility, while Google and Amazon are pursuing small nuclear reactors to power their AI systems.

This highlights a major sustainability challenge for the AI industry. As the demand for AI-powered solutions grows, companies will need to focus on eco-friendly innovations, such as energy-efficient data centers and green cloud providers.

Quantum Computing: The Next Frontier
While AI continues to dominate discussions, quantum computing is quietly advancing in ways that could be even more disruptive. In late 2024, Google unveiled Willow, a quantum chip capable of solving problems that would take supercomputers billions of years, setting the tone for tech trends for 2025.

The Encryption Threat
One major concern with quantum computing is its potential to break current encryption methods, which secure much of the internet’s data. This has led to the creation of initiatives like the Linux Foundation’s Post-Quantum Cryptography Alliance, which aims to develop quantum-resistant encryption algorithms.

Companies need to prepare now for the possibility of “Harvest Now, Decrypt Later” attacks, where encrypted data is stolen today and later decrypted using quantum technology. In 2025, expect to see more organizations adopting quantum-safe cryptography to protect sensitive information.

Tech trends for 2025: Looking AheadFrom the continued growth of AI in creative industries and everyday tasks to the challenges of sustainability and quantum security, 2025 is shaping up to be a year of both innovation and caution.

• For businesses: Keep an eye on AI developments that could enhance efficiency, but stay aware of the legal and ethical implications.
• For individuals: Be prepared for AI to become a bigger part of your life, both in convenience and in the need to spot potential scams.
• For everyone: Quantum computing may feel far off, but its impacts could be closer than we think—especially when it comes to online security.

As we navigate this transformative year, staying informed and adaptable will be key to making the most of these exciting, but complex, technologies.

The post What to Expect: Tech Trends For 2025 appeared first on Semaphore.

View Details

Smoke and sanity testing are two of the most common software testing methodologies. While “sanity testing” is often mistakenly used as a synonym for “smoke testing,” the two focus on different aspects. Understanding their key differences is important to avoid confusion.

Smoke testing involves verifying that the key functionalities of an application still work after a new build. In contrast, sanity testing is more focused, checking that specific changes made to an already validated build work as expected.

In this smoke vs sanity article, you will learn what smoke and sanity testing are, how they work, when to use them, and whether they can be effectively combined.

Let’s dive in!

What Is Smoke Testing?Smoke testing, also called “confidence testing,” “build promotion testing,” or “build verification testing,” is a type of software testing aimed at varying that critical system functionalities work as expected after a new build.

It aims to identify major issues and determine whether the build is stable enough for further testing. As explained in Lessons Learned in Software Testing by Cem Kaner, James Bach, and Brett Pettichord: “The phrase smoke test comes from electronic hardware testing. You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don’t have to do any more testing.”

The practice gained popularity with the advent of Agile development methodologies, where frequent builds are a priority. Smoke testing ensures stability early in the development lifecycle, helping teams decide whether to continue the deployment.

How to Apply Smoke TestingAt its core, smoke testing follows a straightforward process:

Everything starts with the development of a new feature, update, or an entirely new version of the software. Once the development is complete, a new software build is produced and typically sent to a CI/CD pipeline for deployment.

Next, smoke tests are executed. These tests are a selected subset of test cases—including unit tests, E2E tests, API tests, and more. Their goal is to test the most important functionality of the system, ensuring that core features are still working as intended.

If the smoke tests pass, the deployment process continues with more specific testing phases, such as regression or integration tests. On the contrary, if the smoke tests fail, the deployment is halted. The build is rejected and sent back to the development team to address the identified issues. Once fixed, a new build is created, and the cycle repeats until the software passes the smoke testing phase.

In other words, smoke testing is a critical checkpoint in software development. It acts as the first line of defense to ensure that new builds are stable and ready for deployment. This is particularly effective in an automated testing pipeline, as it helps teams quickly understand whether to proceed with deployment or stop it immediately, saving time and resources.

Example of a Smoke Test ScenarioImagine an e-commerce platform where a new payment gateway integration has been added to the checkout process. The development team has modified the application to include that new gateway, which allows customers to choose between multiple payment methods.

The key functionalities that should be verified, regardless of the changes made in an e-commerce application, include:

• Login and sign-up workflow: Ensure that users are still able to log in and create new accounts.
• Cart and checkout workflow: Users still need to be able to add items to the cart, proceed to checkout, and fill in necessary details such as shipping address.
• Payment methods: Check that all payment options are available and function correctly.
• Order confirmation: Ensure that after payment—regardless of the chosen payment method—the order is processed and the user receives a confirmation of the successful purchase.

These smoke tests confirm whether the e-commerce application still works correctly after the new build.

Smoke Testing Frameworks and ToolsThere are no dedicated tools or frameworks designed specifically for smoke testing. Since every software application is unique, the choice of tools depends on the specific functionalities you need to verify in your smoke tests.

For example, in one application, the key functionalities might involve API endpoints, while in another, it could focus on ensuring that the login workflow functions correctly. For testing API endpoints, tools like Postman or Swagger are highly effective. If your smoke tests target core business logic, unit testing frameworks such as JUnit, Jest, or Mocha are well-suited. On the other hand, when smoke testing involves verifying complete user workflows, tools like Playwright, Cypress, or Selenium are ideal choices.

Ultimately, smoke testing is less about the tools themselves and more about the strategy—selecting the appropriate tests to efficiently verify the build’s stability.

Pros and Cons of Smoke TestingLet’s discover the main advantages and disadvantages of smoke testing!

👍 Pros:

• Early detection of critical issues
• Can be run frequently and quickly, as it generally involves only a limited subset of test cases
• Improves build quality
• Can be integrated into CI/CD pipelines for testing automation
• Enhances team productivity by saving time thanks to its shift-left approach to testing

👎 Cons:

• Identifying the right smoke tests requires skills and can be time-consuming, especially in larger projects
• Passing smoke tests can create an illusion of stability, as deeper issues may exist in the application
• Requires a lot of maintenance as the smoke testing suite needs regular updates to reflect changes in core functionality

Wonderful! Time to continue this smoke testing vs sanity testing comparison by exploring sanity testing.

What Is Sanity Testing?Sanity testing, also known as “surface level testing,” is a type of software testing aimed at verifying that specific functionalities or bug fixes in a software build are working as expected. It focuses on checking that minor changes to the application have not introduced unintended issues or caused core features to break.

Sanity testing is a narrow and quick check to confirm that the software is stable enough for more comprehensive testing. In most cases, it is performed after smoke testing.

How to Apply Sanity TestingThe cycle of sanity testing is similar to that of smoke testing. The main difference is that it starts from a more stable build that has already passed smoke testing, rather than from a new build that has yet to be tested.

The main goal of this testing process is to validate changes made to one or a few specific parts of the application. It is a targeted approach that focuses on specific components to ensure they are functioning properly after changes.

Sanity tests perform a quick and superficial check to confirm that the system still works as expected after a modification. Due to its focused and narrow scope, sanity testing tends to be carried out manually by software engineers or QA experts and is often not automated.

Example of a Sanity Test ScenarioConsider again the example of an e-commerce application where the development team has introduced a new payment gateway. After the application has passed smoke testing, sanity tests would focus solely on the feature that has changed—the payment system.

Testers will ensure that all payment methods work as intended, thoroughly validating these features to confirm that the new payment gateway integration functions correctly without affecting other parts of the application.

Sanity Testing Frameworks and ToolsIn most cases, sanity testing does not require dedicated frameworks, as it is achieved manually and without scripts. However, in cases where a more robust and/or automated sanity testing process is needed, the same considerations applied to smoke tests also apply to sanity tests.

Pros and Cons of Sanity TestingLet’s see the main benefits and drawbacks of sanity testing!

👍 Pros:

• Cost-effective, as it is simple and not structured
• Narrow and quick, with focused verification on areas of the application that have changed
• Increases confidence in the robustness of a given build

👎 Cons:

• May require manual effort, as it may not be easily automated
• Effectiveness depends on the tester’s ability (at this exact moment) to accurately identify issues

Can Smoke Tests and Sanity Tests Coexist?TL;DR: Yes, smoke and sanity tests can be used together.

Sanity and smoke tests can coexist in a software development cycle, as they serve distinct but complementary purposes:

Smoke testing is the first step, verifying that core functionalities are working after a new build. Once smoke testing is successful, builds become more stable, and sanity testing is carried out. That focuses on specific changes to ensure those updates did not introduce new issues and bugs.

While both testing approaches can be used together, they are typically applied in different scenarios. Smoke testing is often automated and integrated into CI/CD pipelines, while sanity testing is generally done manually. In a typical process, smoke tests can be automated, and once they pass, manual sanity tests are performed on the changed features to confirm they work as expected.

If you prefer a fully automated approach, you may rely solely on smoke testing or invest in creating automated sanity tests. At the same time, automating sanity tests can be time-consuming and may not justify the effort, as these tests are extremely specific and hard to re-use.

Smoke vs Sanity Testing: Summary TableExplore the differences between the two approaches to testing in the smoke vs sanity comparison table below:

| Aspect | Smoke Testing | Sanity Testing | | --- | --- | --- | | Purpose | Verify that the core functionalities of an application are still working | Check that bugs have been resolved and the introduced changes are functioning as expected | | Goal | Ensures the build is stable enough for further testing | Ensures an application works after changes or fixes to proceed or not with further testing | | When performed | On a new build | On a new already validated build, usually after smoke testing | | Test Execution | Automated or manually | Typically performed manually | | Executed through | A subset of automated test scripts | Usually, manually by testers, developers, or QA experts | | Coverage | Covers core functionalities | Focuses on specific modules where changes were made | | Type of tests | Unit, E2E, integration, etc. (based on the key features of an application) | Depends on the changes to test | | Frameworks and tools | CI/CD tools (Jenkins, CircleCI, Semaphore), API testing (Postman, REST Assured), UI testing (Selenium, Cypress, Playwright), unit testing (JUnit, Jest), custom test automation frameworks, and more | Manual testing tools (TestRail, Zephyr), bug tracking (Jira, Bugzilla), exploratory testing tools, development IDEs for quick verification, and more |

ConclusionIn this smoke vs sanity testing comparison guide, you learned about the differences between the two. While they are often confused and sanity testing is sometimes seen as the same as smoke testing, they are actually distinct methodologies. Both share a similar testing loop, but they are applied at different stages of the development lifecycle.

The post Smoke Testing vs. Sanity Testing appeared first on Semaphore.

View Details

The Security and Compliance program at Semaphore just hit a major milestone – we’ve officially received our SOC 2 Type 2 certification. This means your code and data are even safer with us than ever before. (Not that they weren’t safe before – we’re just proving it now! 🙂)

What This MeansSOC 2 Type 2 compliance means we’ve undergone a thorough, independent audit of our security practices over an extended period. Unlike a point-in-time assessment, this certification verifies that we consistently maintain robust security practices in our day-to-day operations.

The Security You NeedThe certification process examined crucial aspects of our platform:

• Security infrastructure and data protection
• Access controls and authentication systems
• Continuous monitoring and incident response
• System availability and reliability
• Disaster recovery and business continuity

For our customers, this certification provides concrete validation that your code, data, and development processes are protected by proven security measures that meet industry standards.

Why It MattersIf you’re using Semaphore (or considering us), this certification offers several key benefits:

• Simplified vendor security assessments
• Confidence in our security practices and controls
• Independent verification of our operational excellence
• Peace of mind for your security and compliance teams

Our Ongoing CommitmentWhile achieving SOC 2 Type 2 compliance is a significant milestone, we view it as just one step in our ongoing security journey. We remain committed to continuously improving our security posture and maintaining the trust you place in us.

Learn MoreFor those interested in the technical details or requiring our SOC 2 report for compliance purposes, please contact support@semaphore.com. We’re happy to provide more information under NDA.

Thank you to our customers for your trust, and to our team for their dedication to making this achievement possible.

The post Semaphore is Now SOC 2 Type 2 Compliant! appeared first on Semaphore.

View Details

Cloud adoption has outpaced the development of corresponding expertise in cloud operations, leading to a significant skills gap across the industry. In this regard, many organizations have had to abandon cloud initiatives because they lacked the necessary in-house expertise to manage them. As a result, critical projects are delayed, and costs and operational risks increase. Recognizing this gap, Cory O’Daniel co-founded MassDriver, a platform designed to simplify cloud infrastructure management for engineers without extensive cloud or operations experience. In this episode, Cory will explain how MassDriver allows teams to focus on building business value rather than getting bogged down in intricate cloud configurations.

Edited transcriptionAfter working at small and large software development companies, Cory O’Daniel found a widespread lack of knowledge about cloud infrastructure. Looking to fill this gap, he co-founded MassDriver, a platform as a service (PaaS) for cloud DevOps.

As Cory explains, MassDriver came to be from his effort to “make it easier for engineers that may not have this traditional cloud or operations experience to manage the cloud without getting inundated in the eight pages of docs about how to operate RDS, […] how to write Terraform, and how to build your CI pipelines.” In this way, the platform aims to relieve engineers from the burden of setting up their development environments and let them focus on building business value.

What MassDriver doesAccording to Cory, all software companies, whether they realize it or not, have a platform team. Even in cases where the platform is outsourced, he observes that companies are increasingly viewing the platform as an internal product, rather than “something this other team is doing in service for you.” The key distinction, Cory understands, lies in whether they are “consciously building a platform or not.” Considering this platform as the foundation for their development and deployment, Cory understands conscious platform building begins “as soon as you start getting to the cloud, starting to think, ‘how can I make these reproducible, reusable components that my software sits on top of so it makes it easy for me to make new environments or change things out?’ “

Targeting a user base of developers with no operational experience, MassDriver offers a creative system for building and managing complex cloud platforms. Under the hood, MassDriver follows the practice of infrastructure as code (IaC) for managing and provisioning infrastructure by describing it through code. However, MassDriver’s interface allows users to set up infrastructure using diagrams. As Cory explains, “There’s a number of infrastructure-as-code modules that are published into the platform; You can publish your own as well.” Companies can diagram the infrastructure they need and MassDriver will “grab infrastructure as code modules and policy as code modules from the registry and start to provision using that.”

Drawing infrastructure makes it easy to spot the components of cloud environments and understand how they work together. “When you’re in MassDriver”, Cory explains, “you just drag your application out and it does the Terraform for you, does the role assignments, binds it to your Kubernetes workload.”

In this way, MassDriver balances developer flexibility with robust governance. It empowers developers to make architectural changes while abstracting away granular control over credentials, module selection, and tool inputs. MassDriver handles these underlying details, making it ideal for users who have a defined set of architectural requirements but lack the specific skills to implement them.

To achieve this flexibility and extensibility, MassDriver is built on a functional programming model with types, using a type system as its extension mechanism. As Cory explains, “If you’re using, let’s say Mongo Atlas, we don’t have a type for Mongo Atlas in MassDriver by default, but you can push new types into your registry.” Infrastructure code is encapsulated within typed “bundles.” Each bundle defines input types (configuration parameters), output types (results/artifacts), and the underlying infrastructure code, which is treated as a black box. Consequently, MassDriver itself “has no idea what it’s doing”; it simply understands types and their relationships.

Multi-layered testing strategyGiven the complexity of infrastructure deployments, MassDriver employs a multi-layered testing strategy. This strategy addresses the limitations Cory identifies in traditional testing approaches within the DevOps community. The DevOps community needs to be “a bit more test minded,” he explains; “One of the things I think is really missing from our community and the DevOps community is being a bit more test minded.”

Integration testingCory points out that tools like Terraform test often focus on unit testing individual resources, which doesn’t adequately capture the interconnected nature of real-world infrastructure. Testing a single component in isolation, such as a PostgreSQL database, fails to account for crucial dependencies like the VPC and other necessary configurations.

Therefore, MassDriver emphasizes integration testing, which tests different components together to ensure their cohesive functionality within a system—a more accurate reflection of production environments. As Cory suggests, Terraform test is better suited for validating code correctness rather than testing the interactions between different infrastructure components.

Black box testingSince MassDriver treats all components as black boxes, at its core, it tests that inputs conform to the expected types and verifies that the resulting outputs adhere to them and their structures. This simplifies testing by focusing on system behavior rather than internal implementation details.

Provisioning testingOn the other hand, through provisioning testing, MassDriver ensures infrastructure setups function correctly in real environments, validating credential type interactions and compatibility with various published credential types. When infrastructure is provisioned, MassDriver captures the resulting outputs—such as configuration details, connection information, or resource identifiers—as Artifacts. For example, provisioning a database might result in an Artifact containing the connection string, port, and credentials.

Artifacts can then be used by other infrastructure components, creating a modular and reusable system where the output of one module becomes the input for another, effectively managing dependencies. A web application module, for instance, might rely on the database connection details (an Artifact) from a database module. Artifacts also facilitate infrastructure state tracking and versioning, enabling rollback to previous configurations and providing a change history for more robust management. In testing, Artifacts serve as known good outputs to validate the functionality of other components, ensuring that changes in one area don’t negatively impact others.

TerraTest for infrastructure codeTo test their infrastructure’s code, MassDrive uses Gruntworks’ TerraTest, a popular testing framework for infrastructure as code. TerraTest automates the deployment and management of cloud resources within the test suite, verifies that provisioned resources meet expected criteria, and applies load and simulates failure conditions to ensure robustness.

Use-case-driven test casesBecause isolated component testing isn’t feasible, MassDriver uses use-case-driven test cases. Each infrastructure module is designed around specific use cases “that we design each module around,” such as deploying a database in a VPC with specific configurations. These use cases define the required inputs and expected outputs, which are then tested to ensure correct module behavior in real-world scenarios.

Additionally, MassDriver Github has example configurations for each module. For one thing, these examples provide users with pre-configured setups to quickly deploy infrastructure without having to understand every detail. However, these are used as well to test cases and validate the infrastructure modules, ensuring these modules work as expected in various configurations.

Dependencies testingTo test dependencies, MassDriver sets up complete environments with all necessary components, ensuring each piece of infrastructure works correctly within its context. Before testing a specific component, MassDriver uses its own platform to automatically provision required downstream dependencies. For example, testing a PostgreSQL database would first involve setting up the necessary VPC, subnets, and security groups. After provisioning dependencies, MassDriver applies a load to test how well the infrastructure handles real-world usage, such as applying load to a production database to test auto-scaling behavior. By provisioning a full stack of dependent services (e.g., a database, VPC, and application server), MassDriver performs integration tests to verify that all components work together correctly and that changes in one area don’t break the overall system.

Performance optimizationFinally, recognizing that base provisioning can significantly lengthen test completion times, MassDriver employs strategies like parallelizing tasks and adjusting configurations to mitigate this. They also use data approximations for certain tests, such as using smaller data sizes for database restores, to expedite complex operations.

License changes in open sourceOrganizations often begin using open-source software due to its free availability without initial financial commitments. This allows them to experiment, integrate the software into their systems, and assess its suitability before making potential investments or facing licensing restrictions.

In some cases, on the other side, developing the open source software, there are companies looking to eventually profit from it. As Cory puts it, “open source, by and large, is the purchase pathway for many enterprises.” Initially, enterprises often adopt open-source software due to its free availability and ease of initial integration. However, as their reliance on the software grows, they frequently recognize the value of the additional features, support, and stability offered by enterprise or paid versions.

This strategy is frequently employed by companies developing open-source software. The initial goal is to achieve widespread adoption through the free version, establish a user base, and demonstrate the software’s capabilities. Once enterprises become dependent on the software within their critical workflows, these companies can then offer a conversion path to paid, enterprise-grade versions. These paid versions typically include valuable enhancements such as dedicated support, improved scalability to handle larger workloads, enhanced security features, and compliance certifications.

Still, despite these might seem compelling reasons for enterprises to upgrade, many enterprises start by using open-source software for free over an extended period. They integrate it into their workflows, benefit from its functionality, and often delay transitioning to the paid or enterprise version until they are done dealing with the difficulty of managing the open-source version. Eventually, some enterprises decide to pay for the software, either reluctantly because they need additional features, support, or stability, or sometimes willingly when they recognize the value the software brings. However, most of them continue to use the free version.

According to Cory, when “99% of the people that use our thing use it for free, you’re incentivized to change a licensing model.” This pattern of delayed payment creates a situation where open-source software companies rely heavily on converting free users into paying customers. “You’re incentivized to make the open-source version worse,” he concludes. Consequently, companies initially releasing software under an open-source license and later changing that license impair those that have integrated their software into their operations.

As a result, Cory advocates for certain highly prevalent open-source projects to be managed by foundations rather than single organizations. He believes “there’s a point where something becomes so ubiquitous across our industry that it should not be controlled by a single organization.” Placing these projects under the stewardship of a foundation would prevent unilateral license changes and ensure the software remains community-driven, stable, and less susceptible to the whims of a single entity.

Moreover, Cory believes developers should be aware of this situation and ask themselves about the ownership of their tooling: “We should be opening an issue on their GitHub and saying, ‘Why isn’t this a part of the foundation?’ ” In this regard, he also believes there should be more “smaller foundations centered around specific technologies” and with a smaller scope.

DevOps relevanceCory points out that DevOps teams often “don’t get the same budget as developers and are seen as a burden rather than enablers of business value.” He articulates the difficulty in demonstrating their impact, stating that “it’s hard to see what success we add to the business and the value that we create.”

Cory argues that DevOps teams need more budget to acquire platform tooling earlier, rather than building these tools in-house. This budget constraint forces DevOps teams to often rely on free versions of tools for longer periods before justifying or affording paid versions. Since DevOps is responsible for maintaining and optimizing infrastructure, this delay can hinder their operations. They may spend significant time managing and troubleshooting free tools instead of leveraging more robust, supported enterprise versions that could streamline their workflows. Cory suggests that if DevOps teams had the resources to adopt the right tools earlier, they could deliver greater business value by focusing on higher-level tasks rather than managing basic infrastructure issues.

The bottom lineFollow Cory O’Daniel on X at @coryodaniel and Linkedin to learn the latest on platform engineering. You can also listen to his Platform Engineering Podcast.

To learn more about OpenTufo, visit OpenTofu.org and follow @opentofuorg on X and Linkedin.

The post Cory O’Daniel on Infrastructure as Code appeared first on Semaphore.

View Details

Continuous Delivery (CD) transforms how teams ship software by automating the path to production, making releases routine, predictable, and stress-free. With faster delivery cycles and improved quality, CD enables teams to provide users with reliable value.

This guide offers a 10-step checklist for implementing CD effectively, featuring concrete examples, actionable insights, and best practices. Whether you’re starting fresh or refining your processes, this resource will help you build a robust, high-performing delivery pipeline.

The 10-Step Checklist1. Adopt Version Control Best PracticesVersion control is the backbone of modern software development, enabling teams to collaborate effectively and maintain a history of code changes.

• Use Systems Like Git: Implement a distributed version control system like Git to manage your codebase. Git allows multiple developers to work independently and merge their changes seamlessly. It provides robust features for branching, merging, and reverting changes, making it suitable for projects of any size.
• Implement trunk-based development (GitFlow): Establish a branching strategy that aligns with your workflow, such as Gitflow, feature branching, or trunk-based development. Use branches to isolate work on new features or bug fixes.
• Enforce pull requests and code reviews: Incorporate code reviews into your process by utilizing pull requests on platforms like GitHub or Bitbucket. Code reviews enhance code quality, encourage knowledge sharing, and promote team collaboration.
• Maintain clean commit history with conventional commits: Use a standardized commit message format to indicate change type, scope, and purpose, improving readability and traceability. For example, by categorizing commits into types like “fix,” “feat,” or “chore”.

```

1. Create a short-lived feature branch from main (aka trunk)git checkout maingit pull origin maingit checkout -b feature/new-feature# 2. Implement the new feature then stage and commit changesgit add .git commit -m "feat: implement new feature"# 3. Push the feature branchgit push -u origin feature/new-feature# 4. Create a pull request (PR) on GitHub/GitLab/Bitbucket# - Request reviews from team members# - Address any feedback by making additional commits# 5. Once approved, merge your changes back into main (the trunk)# - Optionally rebase or squash before merging to keep history cleangit checkout maingit pull origin maingit merge feature/new-feature# 6. Push merged changes to maingit push origin main

``` 2. Set Up Continuous Integration (CI)Continuous Integration is crucial for detecting issues early and ensuring that code changes integrate smoothly into the main codebase.

• Automate Builds and Tests Using CI Tools: Utilize a CI platform like Semaphore to automate your build and test processes. Semaphore allows you to define workflows consisting of pipelines, which are sequences of tasks to build, test, and deploy your application. You can create pipelines using the visual workflow editor or by writing YAML configuration files for greater control and versioning. For instance, you can set up a pipeline that triggers whenever code is pushed to a branch or a pull request is opened. In your pipeline configuration, you specify the steps needed to build your application and run your test suite.

• Provide Immediate Feedback to Developers: Configure your CI system to report build and test results promptly. Semaphore integrates with version control systems to provide status checks on commits and pull requests. Immediate feedback allows developers to address issues quickly, maintaining code quality and reducing the likelihood of defects reaching production. Semaphore’s integration capabilities ensure that developers are notified of build failures or test issues as soon as they occur, fostering a proactive development environment.

• Implement Automated TestingAutomated testing ensures that code changes do not introduce regressions and that your application behaves as expected.

• Write Unit, Integration, and End-to-End Tests: Develop a comprehensive testing strategy that includes different levels of testing. Unit tests check individual components, integration tests verify the interaction between components, and end-to-end tests simulate real user scenarios. This layered approach helps catch issues at various stages.

• Automate Tests to Run on Every Code Change: Configure your CI pipeline to run tests automatically whenever code is pushed or a pull request is opened. Tools like Semaphore allow you to define jobs within your pipeline that execute your tests. For example, in a JavaScript project using Jest, you can set up a job that runs npm test as part of your pipeline.

Example pipeline configuration for running automated tests on push and pull requests:

name: CI Pipelineon: [push, pull_request]jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Install dependencies run: npm ci - name: Run tests run: npm test - name: Build run: npm run build 4. Manage Dependencies EffectivelyProper dependency management ensures that your application builds and runs consistently across different environments.

• Cache Dependencies: In CI/CD pipelines, caching dependencies can drastically speed up builds. Caches eliminate the need to fetch packages from scratch on every run, saving time and network bandwidth. Ensure the cache invalidates properly when the lockfile changes to avoid stale or inconsistent installs.

Example of caching dependencies in a pipeline:

name: CI Pipeline# Define the default agent (machine and operating system) for all jobsagent: machine: type: e1-standard-2 # Use a standard Linux machine with 2 CPU cores and 4GB RAM os_image: ubuntu2004 # Use Ubuntu 20.04 as the operating system image# Define the sequence of tasks to be executed in the pipelineblocks: - name: Install Dependencies # Descriptive name of the block task: prologue: commands: - checkout # Check out the code from the version control repository - cache restore # Restore cached 'node_modules' directory if available jobs: - name: Install Dependencies # Name of the job within this block commands: - npm install # Install Node.js dependencies specified in package.json epilogue: always: commands: - cache store # Cache the 'node_modules' directory for future pipeline runs * Use lockfiles: Employ tools specific to your programming language to manage libraries and packages. For example, use npm or Yarn for JavaScript projects, pip for Python, or Maven/Gradle for Java. These tools help you specify exact versions of dependencies, ensuring consistency and reproducibility. * Keep Dependencies Updated and Secure: Regularly update your dependencies to benefit from the latest features and security patches. Automate this process using tools like Dependabot or Renovate, which can create pull requests when new versions are available. Additionally, use security auditing tools such as npm audit or Snyk to detect vulnerabilities in your dependencies.

Example dependency management scripts with npm:

{ "name": "my-app", "version": "1.0.0", "scripts": { "audit": "npm audit", "outdated": "npm outdated", "update": "npm update" }, "dependencies": { "express": "^4.18.0", "react": "^18.2.0" }} * npm audit alerts you to known vulnerabilities. * npm outdated shows which packages are behind the latest release. * npm update automatically updates dependencies within their defined semver range.

1. Automate Deployment ProcessesAutomating deployment processes ensures that software releases are consistent, repeatable, and less prone to human error.

2. Utilize Infrastructure as Code (IaC): Implement tools like Terraform, Ansible, or CloudFormation to define and manage your infrastructure through code. IaC allows you to version control your infrastructure configurations alongside your application code, ensuring that environments can be recreated or scaled reliably. By integrating IaC into your CI/CD pipeline, you can automate the provisioning and configuration of environments needed for testing, staging, and production. Semaphore pipelines can include jobs that execute IaC scripts, enabling you to manage infrastructure changes as part of your deployment process.

Infrastructure as Code example using Terraform:

resource "aws_instance" "web" { ami = "ami-0c55b159cbfafe1f0" instance_type = "t2.micro" tags = { Name = "production-web-server" Environment = "production" }} * Maintain Consistent Environments Across Stages: Ensure that your development, testing, staging, and production environments are as similar as possible. Consistency reduces the risk of environment-specific issues and simplifies debugging. Containerization tools like Docker can help achieve consistency by packaging your application and its dependencies into portable containers. In your Semaphore pipeline, you can use Docker containers to run your application in the same environment at every stage. * Enable Zero-Downtime Deployments: An automated deployment strategy such as rolling updates or blue-green deployments can avoid service interruptions. By seamlessly transitioning user traffic between the old and new versions, you ensure uninterrupted service. Tools like Kubernetes or AWS’s Elastic Container Service offer built-in mechanisms for orchestrating zero-downtime rollouts.

By automating your deployment processes and using tools like IaC and Docker, you ensure that each deployment is performed in a controlled and consistent manner. In the next step, we’ll establish a multi-stage pipeline to ensure continuous delivery.

1. Establish Continuous Delivery PipelinesA well-defined pipeline automates the flow of code changes from commit to deployment, ensuring that every change is tested, validated, and ready for release.

2. Define clear stage progression: Outline the stages your code should pass through before reaching production. Common stages include build, test, staging, and production. Each stage can have its own set of checks and balances to ensure code quality and readiness. In Semaphore, pipelines are defined using YAML configuration files. You can specify the sequence of blocks (stages), dependencies, and the conditions under which each block should run.

3. Automate Gating & Promotions: As each stage (e.g., test, staging) completes successfully, you can set up automatic or manual “gates” that determine when to promote the application to the next stage. For instance, after all tests pass in the staging environment, you can automatically trigger a production deployment. This ensures only tested, stable code is promoted.

Multi-stage pipeline example with gating and promotions:

name: CI/CD Pipelineagent: machine: type: e1-standard-2 os_image: ubuntu2004blocks: - name: Build task: jobs: - name: Compile Code commands: - checkout - ./build.sh - name: Test dependencies: - Build task: jobs: - name: Run Unit Tests commands: - ./run_tests.sh - name: Deploy to Staging dependencies: - Test task: jobs: - name: Deploy to Staging commands: - ./deploy.sh staging - name: Deploy to Production dependencies: - Deploy to Staging run: when: "branch = 'master'" task: jobs: - name: Deploy to Production commands: - ./deploy.sh production * Configure Pipelines in Tools Like Semaphore: Use Semaphore’s visual workflow editor or YAML configuration to set up your pipelines. Semaphore allows you to define triggers, conditions, and promotions to control the flow of your pipeline.

1. Integrate Monitoring and LoggingIntegrating monitoring and logging into your CD pipeline ensures that you can track application performance, detect issues early, and respond proactively.

2. Provide a Single Pane of Glass with Unified Dashboards: Consolidate logs, metrics, traces, and alerts into one dashboard for end-to-end observability. Many platforms (e.g., Datadog, Grafana, or Kibana) can bring together data from multiple sources, giving your team an at-a-glance view of system health. This “one pane of glass” approach reduces context-switching and speeds up root-cause analysis.

3. Set Up Centralized Logging and Alerts: Collect logs from all parts of your application into a centralized system using tools like ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, or Fluentd. Centralized logging makes it easier to search, analyze, and correlate logs from different services. Configure alerts to notify your team when certain thresholds are crossed or errors occur. This proactive approach allows you to address issues before they impact users significantly.
4. Utilize Trace Sampling to Optimize Data Storage: Implement distributed tracing with tools like Jaeger, Zipkin, or OpenTelemetry to trace requests across microservices. However, storing every trace can be expensive and unwieldy. Trace sampling helps manage the volume of data by capturing only a fraction of all requests—allowing you to retain critical visibility into complex issues while keeping storage costs in check.
5. Implement Performance Monitoring: Use monitoring tools to track the health and performance of your application in real-time. Tools like Prometheus, Grafana, New Relic, or Datadog can collect metrics on CPU usage, memory consumption, response times, and more. By integrating performance monitoring into your deployment process, you can automatically roll back deployments if certain metrics indicate a problem. Additionally, monitoring helps you understand how new code changes affect the application’s performance.

Example monitoring setup using Prometheus Node.js client:

const client = require('prom-client')const counter = new client.Counter({ name: 'http_requests_total', help: 'Total HTTP requests', labelNames: ['method', 'path', 'status']})app.use((req, res, next) => { res.on('finish', () => { counter.inc({ method: req.method, path: req.path, status: res.statusCode }) }) next()}) By integrating monitoring and logging into your CD pipeline, you enhance the observability of your application, enabling your team to maintain high availability and performance.

1. Ensure Security and ComplianceSecurity and compliance are critical components of a robust Continuous Delivery pipeline. Integrating security practices early and throughout your pipeline helps prevent vulnerabilities from reaching production and ensures that your software meets regulatory requirements.

2. Integrate Security Scans into Pipelines: Incorporate automated security scanning tools into your CI/CD pipelines to detect vulnerabilities in code, dependencies, and configurations. In Semaphore, you can add security scan jobs to your pipelines. For example, you might use tools like Snyk, OWASP ZAP, or SonarQube to scan for known vulnerabilities. By integrating security scans into your pipeline, you ensure that vulnerabilities are identified and addressed before code is deployed.

3. Automate Compliance Checks and Manage Secrets Securely: Automate compliance checks to ensure adherence to industry regulations and standards, such as GDPR, HIPAA, or PCI DSS. Semaphore allows you to manage secrets securely through its Secrets feature. You can store sensitive information like API keys, passwords, and certificates, and inject them into your jobs as environment variables or files without exposing them in your codebase.

Integrating security and compliance into your pipeline reduces risks and helps maintain user trust.

1. Optimize for Performance and ScalabilityOptimizing your application for performance and scalability ensures that it can handle growth and provide a smooth user experience under varying loads. Include performance and load testing in your pipeline to identify bottlenecks and ensure that new code doesn’t degrade performance. Use tools like JMeter, Gatling, Grafana k6, or Locust to simulate user traffic and measure how your application performs under stress.

Design your infrastructure to scale efficiently. Utilize cloud features like auto-scaling, load balancing, and resource allocation. In Semaphore, you can automate the scaling of your infrastructure using IaC tools like Terraform or Kubernetes. Automating infrastructure scaling ensures that your application can handle increased demand without manual intervention.

1. Foster a Culture of Continuous ImprovementThe success of Continuous Delivery relies on the people and culture within your organization. Encouraging continuous learning and improvement ensures that your processes evolve and adapt over time.

2. Hold Regular Retrospectives: Schedule regular meetings for the team to reflect on what is working well and what can be improved. Use insights from your CI/CD pipeline metrics, such as build times and failure rates, to inform discussions. Example topics for retrospectives include pipeline efficiency and bottlenecks, test coverage and reliability, deployment success rates, and feedback from stakeholders

3. Encourage Ongoing Learning and Adoption of Best Practices: Promote a culture of knowledge sharing and professional development. Provide resources for learning, such as training sessions, workshops, and access to relevant literature. Encourage team members to stay updated on industry trends and to experiment with new tools and methodologies.

By fostering a culture of continuous improvement, your team remains agile, innovative, and better equipped to deliver high-quality software.

ConclusionImplementing Continuous Delivery is an iterative process. Start with the basics and gradually enhance your pipeline based on team needs and capabilities. Focus on automation, reliability, and feedback loops to build a delivery system that enables rapid, confident releases.

Here’s a summary of the 10-step checklist:

1. Adopt Version Control Best Practices
2. Implement Automated Testing
3. Set Up Continuous Integration
4. Manage Dependencies Effectively
5. Automate Deployment Processes
6. Establish Continuous Delivery Pipelines
7. Integrate Monitoring and Logging
8. Ensure Security and Compliance
9. Optimize for Performance and Scalability
10. Foster a Culture of Continuous Improvement

Start by evaluating your current processes against this checklist. Identify areas where improvements can be made and prioritize them based on impact and feasibility. Implement changes incrementally, involving your team in planning and execution to ensure buy-in and smooth transitions.

Additional Resources:

• The Continuous Delivery Book
• Google’s DevOps Research
• The Twelve-Factor App
• DevOps Periodic Table
• Semaphore Tutorials:

Ready to take your Continuous Delivery pipeline to the next level? Try Semaphore for efficient and reliable CI/CD workflows. Sign up today and accelerate your path to seamless software delivery.

The post The 10-Step Checklist for Continuous Delivery appeared first on Semaphore.

View Details

2024 has been a landmark year for tech: the AI hype train is at full steam, with new LLM models emerging and Apple finally entering the race. This year has brought a whirlwind of updates: from fresh versions of frameworks and runtimes to monopoly accusations against Google over Chrome. Is 2025 the year we see quantum computers become mainstream? This is the 2024 tech news roundup.

Making AI profitableThe release of ChatGPT in 2022 sparked a chain reaction that continues unabated. With billions of dollars invested in training models, companies are racing to discover sustainable business models.

OpenAI new models
With a valuation of $157 billion but less than $2 billion in revenue, OpenAI is under pressure to generate returns on its massive investments. This year, OpenAI launched GPT-4o and the more advanced o1 model.

Part of this strategy includes introducing new plans, such as the ChatGPT Pro plan, which grants access to GPT-o1’s professional version. Whether the model justifies its $200-per-month price tag remains to be seen (and if that seems steep, don’t even think about a subscription to Devin).

In addition, OpenAI has partially released Sora, the text-to-video AI generator that was teased earlier this year. Sora can create 20-second videos in 1080p without sound, which suggests how compute-heavy AI video generation is.

Apple Intelligence
Apple has been bidding on its time to adopt AI. This year, the company introduced Apple Intelligence, which has full on-device capabilities.

While Apple’s AI offerings remain modest, including improved proofreading, summarization, enhanced Siri suggestions, and image generation, one standout feature has captured attention: the iPad now boasts an official calculator app, and it’s surprisingly impressive.

Meta’s new LlamaMeta released its latest Llama 3.2 large language model in September. The largest models are multimodal, supporting image reasoning, while the smallest are efficient enough to run on modest machines and phones.

The Llama 3.2 family can be downloaded from HuggingFace or llama.com. Llama remains the closest alternative to an open-source LLM, offering a foundation for AI applications beyond mere OpenAI wrappers.

Anthropic Claude can control your machineAnthropic continues to refine its large language models. Its latest innovation, Computer Use, allows Claude to control your device: viewing your screen, moving the mouse, and typing.

This is as amazing as it’s terrifying, but it shows that AI has great potential to automate tasks.

AI blundersWherever you look there’s AI now. It’s on your Google searches, your phones, and your computers, and it’s even made its way into your messaging apps. Under this kind of pressure, not all AI projects have succeeded.

Two notable failures this year were the Rabbit R1 and the Humane Pin. These tech wearables, aimed at replacing smartphones, received harsh reviews and failed to deliver on their promises.

Microsoft also faced backlash with its Copilot Recall feature, which takes periodic screenshots of your computer. While its purpose—to enable an in-device memory you can query—has merit, the feature’s default activation and poor announcement left many users uneasy, evoking Orwellian undertones.

State of DevOps 2024Google released the annual State of DevOps and this year the focus of the survey has fallen predictably on AI.

The report is well worth a read, but this is the TL;DR:

• Increased productivity but not delivery performance: While AI significantly boosts productivity, flow, and job satisfaction, it hasn’t improved software delivery performance as expected.
• Over-reliance concerns: AI adoption correlates with less time spent on meaningful work, suggesting developers may rely too heavily on AI for complex tasks.
• Low trust in AI-generated code: Despite widespread usage, 39.2% of respondents expressed little to no trust in AI-generated code quality. Developers feel compelled to validate and tweak AI outputs, comparing this reliance to the early days of StackOverflow.
• Environmental concerns: Many developers are worried about the environmental impact of AI, citing increased power consumption and resource costs. Microsoft’s decision to reactivate a nuclear facility to power its AI systems has only amplified these fears.
• AI code generation rises: By October, AI-generated code accounted for over 25% of new code at Google.

Developer highlights2024 brought us new runtimes and new versions of popular frameworks.

Deno 2.0Deno hit the 2.0 milestone this year, marking it with an impressive ad. Fully compatible with Node.js, Deno now supports NPM packages, making it an excellent time to give it a spin.

Deno 2.0 also includes:

• A stable standard library
• Support for private NPM registries
• JSR, an alternative package registry for JS and TS modules
• Monorepo support
• A new LTS support channel

Bun on WindowsBun, the latest JavaScript runtime, has finally released a native version for Windows. Previously, Windows users needed WSL to run Bun, but that’s no longer necessary.

Key updates in the latest version of Bun:

• Windows compatibility
• A CSS bundler
• Enhanced NPM compatibility
• Bytecode compilation

React 19 gets a compiler
Following the steps of Svelte, React 19 now includes a new compiler that converts React code into efficient JavaScript, eliminating the need for manual memoization with hooks like useMemo and useCallback.

We’ve covered the new compiler in another post, so here are the highlights:

• Automatic Memoization: The compiler automatically applies optimizations equivalent to useMemo and useCallback, reducing unnecessary re-renders and improving application performance without manual code adjustments.
• Enhanced Performance: Transforming React code into optimized JavaScript can potentially double application performance, leading to faster load times and a more responsive user experience.
• Automatic Code Splitting: The compiler analyzes your application and automatically splits code into optimized bundles, improving load times and ensuring efficient resource utilization.
• Enhanced Tree Shaking: By identifying and eliminating unused code, the compiler reduces the final bundle size, leading to faster application performance.
• Improved Server-Side Rendering (SSR): The compiler optimizes components for SSR, resulting in faster initial page loads and better SEO performance.
• Support for New Syntax and Features: It enables the use of upcoming JavaScript and React features, ensuring your application stays up-to-date with the latest advancements.

Framework releases in 2024This year a lot of popular frameworks got updated:

• Angular 19: introduced experimental zoneless change detection support and server-side rendering improvements
• Laravel 11: brought new features and improvements, including a first-party web socket server called Laravel Reverb, enhancing real-time capabilities for applications.
• Remix 2.10: continued to enhance its full-stack web framework capabilities, focusing on server-side rendering and client-side routing.
• Svelte 5: added Runes, a new signal-powered reactivity API that serves as the core of Svelte 5. This version includes sv, a brand-new CLI to manage Svelte projects.
• Vue.js 4: introduced a new Composition API, improved TypeScript support, and enhanced performance optimizations.
• Django 4.2: continued to build upon its high-level Python framework, promoting rapid development and clean design, with updates that enhance scalability and security for web applications.
• Ruby on Rails 8: introduces several significant features and enhancements aimed at empowering developers to deploy and manage applications without relying on the cloud.
• Next.js 15: introduces React 19 support, new APIs, turbo pack improvements, and improvements on caching.

Google continues to make the headlinesGoogle remains at the forefront of AI innovation while simultaneously grappling with regulatory challenges. You may recall that Google was the original creator of transformers, the foundation for ChatGPT and all LLMs.

AlphaProof gets a medalGoogle continues its advancements in deep learning. In 2016, Google’s AlphaGo defeated Lee Sedol in Go, the then-world champion in the sport. This year, the Alpha team focused on maths. AlphaProof and AlphaGeometry AI systems competed at the International Mathematical Olympiad, solving 4 out of 6 problems and winning the equivalent of a silver medal.

New quantum computer chipA few weeks ago, Google announced another breakthrough in quantum computing, with a new quantum chip named ‘Willow’. The new chip benchmarks are impressive: it can complete Random Circuit Sampling (RCS) tasks in under five minutes, something that would take the fastest classical supercomputers billions of years.

Google browser monopolyOn a more controversial note, U.S. regulators found Google guilty of engaging in monopoly practices related to its search and browser products. The U.S. government is pushing Google to sell Chrome, which currently holds a 67% global market share. If that happens, which company has deep enough pockets to buy a browser valued at $20 billion?

ConclusionAs the relentless march of AI continues, we are witnessing the tech industry reach new levels of maturity. Developers are making clear decisions about where they want AI—and where they don’t. Let’s hope 2025 brings a more balanced approach to AI integration.

As we look forward to another year, one thing is certain: change. It’s an exciting time to be part of the developer community. Here’s to another year of building, learning, and innovating—happy coding!

The post 2024 Tech Roundup: The News You Might Have Missed appeared first on Semaphore.

View Details

Despite the pervasiveness of open-source software, its sustainability is increasingly under threat. While some communities have established robust ownership for distributing and maintaining their software, others, despite their relevance and popularity among enterprises, are lacking adequate support. In this episode, Tembo.io Head of Education Floor Drees explains the need for a more collaborative and sustainable approach to open-source development. She will help us explore the current state of the Postgres ecosystem and the broader open-source community to point out the critical issues that need to be addressed to ensure the future of these vital technologies.

Edited TranscriptionA series of notable positions across influential tech companies characterize Floor Drees’ career. From Microsoft to Grafana Labs, and later Aiven, Floor gradually specialized in database technologies. Currently, she is the head of education at Tembo, a developer platform built on Postgres (also known as PostgreSQL) designed as a unified solution to employ data services and build database applications through its over 200 Postgres extensions.

Trusting Postgres extensionsAs a reference in the Postgres ecosystem, Tembo provides a trust-worthy interface for employing extensions otherwise scatteredly available. “It is a bit of the Wild Wild West,” Floor says, with “a lot of different extension authors hosting their extensions in very different places.”

The issue of decentralization goes further than a mere inconvenience: Each source implies different hosting and development practices to be understood, adapted, and trusted separately. For organizations considering implementing new extensions, the process is fraught with uncertainty. As Floor notes, companies become “super cautious and anxious to adopt” extensions due to the unclear provenance and reliability of these add-ons.

Unlike ecosystems like RubyGems, which offers transparency and credibility through information about downloads, contributors, and companies involved, Postgres extensions lack a comprehensive trust framework. Floor believes it is “very difficult to understand which are the tools that you can trust, rely on, comfortably put into your stack and forget about to a certain extent.”

The culprit behind Postgres’ extensions issue is the database’s own development philosophy. Unlike most open-source projects, Postgres adopts a more closed approach to contributions characterized by a steep learning curve. In this sense, contributors need to actively participate in mailing list discussions before proposing and designing changes. As a result, there’s a limited number of merge requests, which undergo rigorous review and are typically incorporated into yearly releases.

In like manner, Postgres does not use Github for contributing, as it “incentivizes sort of drive-by contributions,” says Floor, while demanding reviewing many pull requests. However, she points out that Postgres’ decision implies that “growing your contributor base with new people that might bring new perspectives is going to be harder.”

Moreover, extension creators have a diminished role in comparison with core developers. In Postgres, unlike other projects like Ruby, only those contributing to its core are considered contributors and listed as such on its website. Floor understands this is an unfair perspective, as she believes “extensions, sorting of the tooling, and having a healthy ecosystem is what makes a project grant.” “Not recognizing those types of contributions, I find that unfortunate. That’s the word I’m gonna choose, unfortunate,” she says.

The future of open source: Responsibility, legislation, and community sustainabilityBeyond Postgres, the overall open-source ecosystem demands a fundamental reimagining of how its communities collaborate and sustain themselves. At the heart of the issues is the increasingly uneven contributions, marked by a stark disparity between individual developers and massive tech companies. The situation is particularly precarious for smaller contributors. Many extensions are maintained by individual developers, even though they might support critical infrastructure used by major corporations.

As a way to achieve economic sustainability, open-source projects have started re-licensing their products to various degrees; for example, last year data platform Directus started charging large enterprises for production licenses. Floor is concerned about the disruptive potential of sudden licensing shifts and the “work and costs associated with switching out technology”: “You don’t necessarily see those coming and you need to really quickly find an alternative for potentially a really vital piece of your product to sort of like switch it out.” “Those are mostly larger projects,” she explains, “but also smaller projects too changed their license because they’re just done with bigger companies freeloading on their work.” Consequently, Floor believes more and more projects will move under the influence of foundations and, to avoid security issues or license changes, “companies will probably look at foundation-backed projects.”

The security concerns behind the sustainability issue have led governments to take notice of it. This year, the European Union’s Cyber Resilience Act (CRA) entered into force over software and hardware manufacturers and retailers to regulate and mandate cybersecurity requirements, such as reporting vulnerabilities, providing software updates, and auditing products.Floor believes the CRA is an overall step in the right direction that “might force —for lack of a better word— organizations to care a lot more” and feel “an obligation to put more resources” on the software they rely on as “they now have the obligation to go back and contribute their fixes to these projects.” “They can’t just fix it for their customers anymore; that is not allowed,” she concludes.

The bottom lineFor learning about Postgres extensions, check Tembo’s extension registry Trunk Visit Floor’s website to learn more about her work. You can follow her on:

X at @DevOpsBarbie.

Bluesky at@floord.bsky.social.

Mastodon at @floord@hachyderm.io.

The post Floor Drees on Building Sustainable Open-Source Ecosystems appeared first on Semaphore.

View Details

Yes, I already know: when developers read the word “testing,” a mix of emotions comes to their mind: fear, anxiety, frustration…

For this reason, this post aims to explain what test automation is and how developers can benefit from it, in terms of productivity and code quality.

In particular, it discusses what unit testing and integration testing are, how they differ from each other, and how to implement them in Python using Flask, by providing a step-by-step tutorial.

So, fear no more, and let’s dive right into it!

What is Test Automation?Test automation is the process of automating the testing activities developers do to ensure that their code works as expected. It involves writing scripts or programs that simulate user interactions with an application: this helps verify its behavior under different conditions, and report any issues found.

Manual VS Automated TestingManual testing is the first step developers need to overcome when they learn to code if they want to ensure their code works as expected when users will use it.

This process needs total human intervention, which means that every time you perform a new test, you’ll need to manually execute each step involved in your test case. So, as understandable, this can be very time-consuming and prone to errors due to human mistakes.

Also, the process of manual testing is generally very simple. For example, a typical use case in Python is to write some print() here and there along the code to check whether everything works fine. But, indeed, this does not ensure the code is working correctly: it only shows that the code runs without crashing until a specific point.

On the other hand, automated testing—while automating the process of testing—also provides processes and procedures that actually ensure the code works as expected.

Benefits of Test AutomationSo, let’s point out some of the benefits of using automated testing:

• Standardization: Test automation equips developers with a standardized approach to testing by providing frameworks and processes that actually ensure the code works as expected.
• Reduced human errors: As with every automation, test automation eliminates the human errors that can occur in the processes.
• Increased code quality: By ensuring the code works as expected, test automation helps improve the overall quality of the code. In fact, test automation facilitates consistent and thorough testing: this reduces the number of bugs entering into production, thus improving the overall quality of the code.
• Faster code delivery: With automated testing, developers can quickly identify and fix issues before releasing software updates. This speeds up the development cycle and allows teams to deliver high-quality products more frequently.
• Increased test coverage: The goal of testing software is to ensure that all parts of the code are tested thoroughly (or, at least, the majority of them). While manual testing may miss certain areas or scenarios, automated tests can cover a wider range of cases and edge situations, increasing confidence in the reliability of the system when delivered to production.

Test Automation: Understanding Unit TestsIn the context of test automation, we can define a “unit” as the smallest testable piece of a software application. This means that a unit could be a function, method, class, module—or anything related—depending on how granular you want to get or depending on the software you are developing.

So, unit testing is the test automation practice that focuses on testing individual components of a program to ensure they work as intended, separately from the other parts of the same program. For this reason, this is generally the first level of testing performed during the development process.

For example, in a web application that manages online purchasing after users have logged in, a unit test can focus on the login phase, ensuring the login credentials are validated properly, and that the user is redirected to the correct page upon successful authentication.

While this testing practice does not guarantee that the entire program works as expected, it ensures that each component behaves correctly independently from the others. For this reason, this does not mean that unit tests are unuseful; instead, they provide some benefits during the software lifecycle like:

• Early bug detection: Unit testing is often practiced by developers themselves while coding. This allows them to identify and fix bugs easily during the development phase. This early detection allows addressing issues before they become more complex and, eventually, costly.
• Enabling frequent releases: When unit tests are implemented effectively, they help ensure that changes made to one part of the codebase don’t break other parts. This makes it easier for teams to make frequent releases without worrying about introducing new issues or regressions.
• Simplified refactoring and maintenance: Well-written unit tests act as documentation for the codebase, making it easier for developers to understand how different parts interact. This serves as safety nets, helping teams against unintended side effects when refactoring or modifying existing functionality during maintenance tasks.

Characteristics of Effective Unit Tests: The FIRST PrincipleDue to the nature of unit testing, effective unit tests should follow the so-called FIRST principle:

• Fast: Staring at the screen during the first of 10 unit tests running, knowing the first will take 15 minutes alone to complete, is not acceptable. Instead, unit tests should run quickly, ideally within milliseconds. This enables developers to run them frequently during the development process, allowing for rapid feedback and iteration. Fast tests, in fact, encourage developers to run them frequently, integrating them seamlessly into the development workflow with the aim to catch defects early and accelerate the development cycle.
• Independent: Each unit test should be independent of other tests, meaning they can be executed in any order without affecting the results. This promotes parallel execution, teamwork, and efficient test suites, which is a way to speed up the testing process.
• Repeatable: If you run a unit test today obtaining a different result than yesterday, something went wrong. Instead, unit tests should produce consistent results regardless of the environment or machine where they’re run. This ensures reliable outcomes across different setups.
• Self-validating: A good unit test should use assertions to validate outcomes, providing a clear pass or fail based solely on its own logic result without requiring external input, interpretation, or manual intervention.
• Timely: Ideally, tests should be written just before the production code that makes them pass. This enables the test to guide your design choices, aligning with the so-called Test-Driven Development (or TDD). To get the idea, consider that for some developers “timely” means writing tests in the same timeframe as the code itself.

Test Automation: Understanding Integration TestsIntegration testing is another important aspect of test automation that focuses on testing the interactions between multiple units of a software application. So, unlike unit tests, integration tests verify how these components work together to achieve the desired outcome as a whole.

The main idea behind integration testing is to simulate real-world usage scenarios and detect potential issues that arise when combining different parts of the application. These tests help uncover problems such as data inconsistencies, communication failures, or unexpected behaviors caused by interactions between various components.

As an example, let’s consider the previously-mentioned web application scenario. In this case, an integration test might involve simulating the entire purchasing flow, including logging in, selecting items, adding them to the cart, proceeding to checkout, and completing payment. Such a comprehensive test ensures that all aspects of the purchase process function correctly when integrated, allowing deployment to production with greater confidence.

Integration Testing MethodsThere are several methods used for integration testing, each with its own advantages and trade-offs. Here are some common approaches:

• Big-bang: This testing method involves integrating all units simultaneously and testing them as a complete system. While this is a thorough method, it can lead to longer test cycles since fixing bugs discovered later in the process requires retesting earlier stages. For this reason, this method is typically used when the components are relatively independent and can be tested individually.
• Top-down: Top-down integration starts with higher-level units and gradually integrates with lower-level ones. This approach helps isolate issues closer to the top levels but may overlook deeper-seated problems in lower layers. So, while this method is highly effective in the early detection of interface issues and major control points, it requires the creation of mock objects for lower-level units, which can be complex and time-consuming.
• Bottom-up: Bottom-up integration tests start with lower-level units and progressively integrate them into higher-level ones. This method is useful for identifying issues related to internal interfaces and dependencies among components. Its great advantage is that it allows developers to test the lower-level modules and their interactions before integrating them into the higher levels, generally with no need for mock objects. However, it delays the testing of the higher-level functionalities, which may push back the identification of defects, slowing down the development process.
• Incremental testing: Incremental testing combines the elements of the other testing approaches. It begins by testing smaller units and gradually increases complexity by integrating additional components over time when the tested ones work as expected. This method strikes a balance between early defect detection and timely completion of higher-level tests, offering flexibility and adaptability throughout the development lifecycle. However, it is time-consuming and may not be suitable when there’s the need to match tight deadlines.

Key Differences Between Unit and Integration TestingNow that we’ve covered both unit and integration testing, let’s highlight the key differences between them to provide a clearer overview:

• Scope and focus: Unit tests concentrate on testing individual units, whereas integration tests examine how those components interact with each other. So, in other words, unit tests focus on ensuring each component functions correctly, while integration tests ensure that the combined components behave as expected as a whole.
• Isolation vs interaction: Unit tests operate in isolation, focusing on a single unit’s behavior without considering its interactions with other components. On the contrary, integration tests evaluate how units collaborate and communicate with each other, taking into account their relationships and dependencies. To do so, they require mocking or stubbing techniques to simulate the behavior of dependent components, as users supposedly will use them.
• Complexity and execution time: Unit tests tend to be simpler and faster to execute compared to integration tests. This is because unit tests usually involve fewer steps and less setup, while integration tests require setting up a more realistic environment that mimics actual usage scenarios.
• Debugging and failure analysis: Debugging unit tests is generally straightforward, as they target isolated components. On the other hand, debugging integration tests can be more challenging due to the increased complexity and interdependencies between components. This is why failure analysis in integration tests may involve tracing through multiple interconnected units to pinpoint the root cause of issues.
• Actors involved: While this topic depends on the project structure and organization, we can say that unit tests are primarily conducted by developers who create and execute them to ensure the proper functioning of individual units. Instead, as integration tests are more complex, they are generally performed by QA teams; yet, it may involve collaboration between developers and testers, as well as stakeholders representing end-users’ perspectives.

Test Automation: A Step-by-Step Python Tutorial Using FlaskAfter all this theory, it’s now time to get’s hands on code!

In this section, you will learn how to create a simple Flask application and how to test it. For the sake of simplicity, the application can be tought of as an online calculator that performs basic arithmetical operations. Specifically, it adds and multiply numbers.

But before diving into the code, let’s start by listing all you need to correctly set up your environment.

Prerequisites, Requirements, and Repository StructureBefore you begin, make sure you have Python 3.8+ installed on your computer.

Then, create a repository – I named it flask_app – with the following structure:

├── app.py├── templates/│ ├── index.html│ └── result.html└── tests/ ├── __init__.py ├── test_unit.py └── test_integration.py The venv/ folder contains a virtual environment. You can create it by typing:

python3 -m venv venv To acvitate it on Windows, type:

venv\Scripts\activate And on Linux/MacOS, type:

source ./venv/bin/activate After the virtual environment has been activated, install the required packages by typing:

pip install Flask pytest requests Now you are ready to write your code!

Building The ApplicationNow you can create your online calculator by writing the following code inside app.py:

from flask import Flask, render_template, request, redirect, url_for, jsonifyapp = Flask(__name__)def add_numbers(a, b): """Adds two numbers and returns the result.""" return a + bdef multiply_numbers(a, b): """Multiplies two numbers and returns the result.""" return a * b@app.route('/')def index(): """Render the main page with forms.""" return render_template('index.html')@app.route('/calculate', methods=['POST'])def calculate(): """Handle form submission and display the result.""" operation = request.form.get('operation') a = request.form.get('a') b = request.form.get('b') c = request.form.get('c') try: a = float(a) b = float(b) # Perform the selected operation if operation == 'add': result = add_numbers(a, b) return render_template('result.html', result=result) elif operation == 'multiply': result = multiply_numbers(a, b) return render_template('result.html', result=result) elif operation == 'add_multiply': c = float(c) sum_result = add_numbers(a, b) result = multiply_numbers(sum_result, c) return render_template('result.html', result=result) else: return render_template('result.html', error="Invalid operation selected.") except (TypeError, ValueError): return render_template('result.html', error="Invalid input provided.")@app.route('/add', methods=['GET'])def add(): """API endpoint to add two numbers.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) result = add_numbers(a, b) return jsonify({'result': result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400@app.route('/multiply', methods=['GET'])def multiply(): """API endpoint to multiply two numbers.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) result = multiply_numbers(a, b) return jsonify({'result': result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400@app.route('/add_multiply', methods=['GET'])def add_and_multiply(): """API endpoint to add two numbers and then multiply the result by a third number.""" try: a = float(request.args.get('a')) b = float(request.args.get('b')) c = float(request.args.get('c')) sum_result = add_numbers(a, b) final_result = multiply_numbers(sum_result, c) return jsonify({'result': final_result}) except (TypeError, ValueError): return jsonify({'error': 'Invalid input'}), 400if __name__ == '__main__': app.run(debug=True) So, this code created an app that does the following:

• Adds two numbers (with the add function).
• Multiply two numbers (with the multiply function).
• Creates three endpoints that can be usable as a web app. One endpoint sums numbers (the add endpoint), one multiply numbers (the multiply endpoint), and the last one adds two numbers and multiplies the result by a third number (with the add_multiplyendpoint). Each endpoint manages the expected data types and errors, in case of bad inputs.

To make it locally working, you also need to write the following code into the index.html file:

Calculator App Calculator App Add Two Numbers First Number (a): Second Number (b): Add Multiply Two Numbers First Number (a): Second Number (b): Multiply Add Two Numbers and Multiply the Result First Number (a): Second Number (b): Multiplier (c): Calculate This will manage the UI of the app.

Also, you need to add the following code into the result.html file:

``` Calculation Result Calculation Result {% if result is defined %} The result is: {{ result }}

{% elif error is defined %}        Error: {{ error }}

{% endif %}    Perform another calculation

``` This will manage the UI of the result page.

Now you can run the app by typing:

python app.py and open your browser at http://localhost:5000/ to see it live.

You are now ready to test it!

Manual TestingFor the sake of completeness, let’s briefly discuss how to manually test this application.

For each of the three endpoints, you should manually:

• Verify all the possible acceptable cases. This means testing numbers, which means: integers and floating (as well as negative numbers, null values, etc…).
• Verify all the possible unacceptable cases. This means testing strings, which means: letters, symbols, etc. This also means that the application should return an error message to say that the input is invalid, not break the code.

For brevity, let’s just manually verify one endpoint. For example, let’s test the add one.

In the positive case, you should insert two numbers:

When clickinc on add, you should be redirected to another page showing the result of the addiction:

If you now insert a letter instead of a number, you should receive an error message:

As understandable, even in the case of a basic app, the effort when manually testing code takes a lot of time and resources.

So, let’s now use automated testing to save our time!

Unit TestsTo create automated tests based on unit testing, you can consider the add_numbers() and multiply_numbers() functions to be units. In this scenario, you can write the following code into the test_unit.py in the tests/ folder:

import unittestimport sysimport os# Add the parent directory to sys.path to import app.pysys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))from app import add_numbers, multiply_numbersclass TestMathFunctions(unittest.TestCase): """Unit tests for math functions with broad test cases.""" def test_add_numbers_valid(self): """Test the add_numbers function with valid numeric inputs.""" # Test with positive integers self.assertEqual(add_numbers(1, 2), 3) # Test with negative integers self.assertEqual(add_numbers(-1, -1), -2) # Test with zero self.assertEqual(add_numbers(0, 0), 0) # Test with positive floats self.assertEqual(add_numbers(1.5, 2.5), 4.0) # Test with negative floats self.assertEqual(add_numbers(-1.5, -2.5), -4.0) # Test with mixed integer and float self.assertEqual(add_numbers(1, 2.5), 3.5) # Test with very large numbers self.assertEqual(add_numbers(1e20, 1e20), 2e20) # Test with very small numbers self.assertAlmostEqual(add_numbers(1e-20, 1e-20), 2e-20) def test_add_numbers_invalid(self): """Test the add_numbers function with invalid (non-numeric) inputs.""" # Test with strings (letters) with self.assertRaises(TypeError): add_numbers('a', 'b') # Test with strings that look like numbers with self.assertRaises(TypeError): add_numbers('1', '2') # Test with None with self.assertRaises(TypeError): add_numbers(None, 2) # Test with lists with self.assertRaises(TypeError): add_numbers([1, 2], 3) # Test with dictionaries with self.assertRaises(TypeError): add_numbers({'a': 1}, {'b': 2}) # Test with boolean values with self.assertRaises(TypeError): add_numbers(True, False) def test_multiply_numbers_valid(self): """Test the multiply_numbers function with valid numeric inputs.""" # Test with positive integers self.assertEqual(multiply_numbers(2, 3), 6) # Test with negative integers self.assertEqual(multiply_numbers(-2, -3), 6) # Test with positive and negative integers self.assertEqual(multiply_numbers(-2, 3), -6) # Test with zero self.assertEqual(multiply_numbers(0, 100), 0) # Test with positive floats self.assertEqual(multiply_numbers(2.5, 4), 10.0) # Test with negative floats self.assertEqual(multiply_numbers(-2.5, -4), 10.0) # Test with mixed integer and float self.assertEqual(multiply_numbers(3, 0.5), 1.5) # Test with very large numbers self.assertEqual(multiply_numbers(1e10, 1e10), 1e20) # Test with very small numbers self.assertAlmostEqual(multiply_numbers(1e-10, 1e-10), 1e-20) def test_multiply_numbers_invalid(self): """Test the multiply_numbers function with invalid (non-numeric) inputs.""" # Test with strings (letters) with self.assertRaises(TypeError): multiply_numbers('a', 'b') # Test with strings that look like numbers with self.assertRaises(TypeError): multiply_numbers('1', '2') # Test with None with self.assertRaises(TypeError): multiply_numbers(None, 2) # Test with lists with self.assertRaises(TypeError): multiply_numbers([1, 2], 3) # Test with dictionaries with self.assertRaises(TypeError): multiply_numbers({'a': 1}, {'b': 2}) # Test with boolean values with self.assertRaises(TypeError): multiply_numbers(True, False)if __name__ == '__main__': unittest.main()

NOTE: The use of the sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) is to allow importing from the parent directory (..) where the app.py file resides. This is necessary because the current script is located within the tests/ subdirectory, and thus needs to navigate upwards to access files outside its immediate scope.

As you can see, with only a few lines of code, this test file takes care of different possibilities like inserting negative numbers, very big numbers, letters, and more.

Now, if you go into the tests/ folder and launch the test_unit.py file by tiping python3 test_unit.py you should receive an output like this one:

Ran 4 tests in 0.001sFAILED (failures=2) So, in this case, you have two tests failed. Why has this happended? Return up to the previous code; as you can see, it reports various functions that represent different cases; let’s consider two of them:

• The test_add_numbers_valid() function creates and tests positive cases for the add_numbers() function. This means that the values tested are acceptable ones.
• The test_add_numbers_invalid() function, on the other hand, creates and tests negative cases for the add_numbers() function. This means that the values tested are not acceptable ones.

So, a question may arise now: have you managed unacceptable values in the functions in the app.py file? Well, the answer is no! In fact, if you scroll the terminal, you should see an output like this one:

AssertionError: TypeError not raised This happens because the add_numbers() and multiply_numbers() functions in the app.py are not raising a TypeError when provided with invalid (non-numeric) inputs, as your tests expect.

Hooray! Here’s another big result achieved! Not only you can test multiple scenarios automatically, but you can also find ways to improve your code. This is why the Test-Driven Development approach is so powerful!

So, if you want to improve your code in the app.py by also considering to raise a TypeError with invalid inputs, you can modify the functions like so:

def add_numbers(a, b): """Adds two numbers and returns the result.""" if isinstance(a, bool) or isinstance(b, bool): raise TypeError("Boolean values are not allowed") if not isinstance(a, (int, float)) or not isinstance(b, (int, float)): raise TypeError("Both inputs must be int or float") return a + bdef multiply_numbers(a, b): """Multiplies two numbers and returns the result.""" if isinstance(a, bool) or isinstance(b, bool): raise TypeError("Boolean values are not allowed") if not isinstance(a, (int, float)) or not isinstance(b, (int, float)): raise TypeError("Both inputs must be int or float") return a * b At this point, you should get no more errors and the result should be something like this one:

Ran 4 tests in 0.001sOK Which means that 4 test have been performed in 0.001 seconds and everything went fine!

Integration TestingLet’s now use a botton-up approach to perform integration testing. In this scenario, you can write the following code into the integration_unit.py in the tests/ folder:

import unittestimport sysimport os# Add the parent directory to sys.path to import app.pysys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))from app import appclass TestAppIntegration(unittest.TestCase): """Integration tests for the Flask application.""" def setUp(self): """Set up the test client.""" # Configure the app for testing app.config['TESTING'] = True self.client = app.test_client() def test_add_endpoint(self): """Test the /add endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/add?a=10&b=20') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 30.0}) # Invalid input test (non-numeric values) response = self.client.get('/add?a=foo&b=bar') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'b') response = self.client.get('/add?a=10') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_multiply_endpoint(self): """Test the /multiply endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/multiply?a=5&b=4') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 20.0}) # Invalid input test (non-numeric value for 'b') response = self.client.get('/multiply?a=5&b=bar') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'b') response = self.client.get('/multiply?a=5') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_add_multiply_endpoint(self): """Test the /add_multiply endpoint with valid and invalid inputs.""" # Valid input test response = self.client.get('/add_multiply?a=2&b=3&c=4') self.assertEqual(response.status_code, 200) self.assertEqual(response.get_json(), {'result': 20.0}) # (2 + 3) * 4 = 20 # Invalid input test (non-numeric value for 'a') response = self.client.get('/add_multiply?a=foo&b=3&c=4') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) # Missing parameter test (missing 'c') response = self.client.get('/add_multiply?a=2&b=3') self.assertEqual(response.status_code, 400) self.assertEqual(response.get_json(), {'error': 'Invalid input'}) def test_calculate_route(self): """Test the /calculate route for form submissions.""" # Test addition via form submission response = self.client.post('/calculate', data={ 'operation': 'add', 'a': '5', 'b': '7' }) self.assertIn(b'The result is: 12.0', response.data) # Test multiplication via form submission response = self.client.post('/calculate', data={ 'operation': 'multiply', 'a': '4', 'b': '6' }) self.assertIn(b'The result is: 24.0', response.data) # Test add and multiply via form submission response = self.client.post('/calculate', data={ 'operation': 'add_multiply', 'a': '2', 'b': '3', 'c': '5' }) self.assertIn(b'The result is: 25.0', response.data) # Test invalid input via form submission response = self.client.post('/calculate', data={ 'operation': 'add', 'a': 'foo', 'b': 'bar' }) self.assertIn(b'Error: Invalid input provided.', response.data)if __name__ == '__main__': unittest.main() And you chould obtain the following result:

Ran 4 tests in 0.023sOK So, here’s what this code does

• Validates end-to-end functionality: It ensures that the application handles requests properly from the client’s perspective, including processing inputs, performing computations, and returning appropriate responses for each endpoint.
• Tests interactions between components: Verifies that the individual functions (add_numbers, multiply_numbers) integrate correctly with the Flask routes and handle data as expected.
• Checks input validation and handles errors: Confirms that the application responds correctly to invalid inputs or missing data, returning appropriate error messages and HTTP status codes.
• Simulates real user behavior: Uses the Flask test client to mimic real-world interactions without running the actual server, facilitating efficient and automated testing.

Note that, with this code, you tested all the endpoints and their expected functionalities from a user’s perspective. The add_multiplyendpoint is particularly to be mentioned because it does not perform its logic independently from the others two, so it couldn’t be tested with only unit tests (if you’d write a unit test for it, you’d end up testing individual components – add_numbers() or multiply_numbers() – in isolation.)

ConclusionsIn this article, we presented the basics of test automation, discussing both unit testing and integration testing approaches.

While the theory is important, understanding these concepts requires practical experience and time. For this reason, the step-by-step guide provided offers a hands-on approach to learning about test automation using Python and Flask. But this is only the first step towards mastering this topic, so keep practicing and exploring the various techniques to become proficient in test automation!

Also, for improving even more your experience, you find all the code in this public repository. When you clone it, consider that you can do even more. In fact, to improve your automated testing skills and experience even more, I have created a CI using Semaphore CI: this firse up your automation journey!

Happy coding!

The post Unit Testing vs. Integration Testing: Test Automation Basics appeared first on Semaphore.

View Details

When it comes to testing, Test-Driven Development (TDD) and Behavior-Driven Development (BDD) are two of the most widely used methodologies. While the terms “TDD” and “BDD” are often used interchangeably, they represent distinct approaches involving different goals and stakeholders. It is important to understand the key differences between them to avoid confusion.

TDD focuses on complete coverage of code functionality by writing tests before the actual code implementation. In contrast, BDD is about capturing and validating business requirements through tests written in natural language, making them accessible and understandable to all stakeholders.

In this TDD vs BDD article, you will learn what Test-Driven Development and Behavior-Driven Development are, how they work, when to use each, and whether they can be combined effectively.

Let’s dive in!

What Is TDD (Test-Driven Development)?TDD, short for Test-Driven Development, is a proven software development methodology where tests are written before the actual code. The technique follows a straightforward cycle:

1. Write a test for a new feature
2. Create the minimum code needed to pass the test
3. Refactor the code while ensuring the test still passes

This approach promotes better design, enforcing code correctness, and leads to higher-quality software.

TDD is attributed to Kent Beck, who introduced it in the late 1990s as part of Extreme Programming—an Agile methodology aimed at improving software quality and the development process.

Over time, TDD gained widespread popularity, especially within Agile development. Today, it is a cornerstone of modern software engineering, particularly in environments that prioritize automated testing.

How to Apply TDDAt its core, Test-Driven Development (TDD) follows a repetitive cycle:

The cycle in the above diagram is often summarized as Red-Green-Refactor. This process begins with the essential initial step of creating a list of test cases. From this list, you need to select one test and apply the Red-Green-Refactor cycle to it. Once completed, move to the next test in the list.

Note: Prioritizing test cases in the right order is key to efficiently addressing the most critical points in the design.

The loop is repeated for all test cases until the desired functionality is fully implemented. By breaking development into these manageable steps, TDD guides iterative and controllable progress.

Now, let’s explore the three main steps of the cycle in detail: Red, Green, and Refactor.

Step #1: Red PhaseThe cycle begins by writing a test that defines the next bit of functionality. This phase is called “Red” since the test must fail, as the corresponding code does not exist yet.

Benefits:

• Provides a clear, incremental development goal
• Forces you to focus on requirements and expected behavior before implementation

Potential issues:

• Writing meaningful tests without the corresponding code can be challenging
• Poorly structured tests may lead to ambiguity or misdirection in the next phase

Step #2: Green PhaseThe objective of this phase is to write just enough code to make the test pass. That code does not have to be perfect or optimized—it just needs to work.

Benefits:

• Encourages minimalism, avoiding over-engineering
• Quickly validates functionality, giving confidence in the code

Potential issues:

• Focusing solely on passing the test might result in rushed, unstructured code
• Tests with unclear objectives may lead to flawed implementations

Step #3: Refactor PhaseAfter the test passes, you need to improve the structure and quality of both new and existing code without altering the functionality under test.

Benefits:

• Leads clean, maintainable, and extensible code
• Reduces technical debt by addressing redundancies and inefficiencies

Potential issues:

• Developers may skip this phase, leading to messy, hard-to-maintain code
• Poor refactoring can introduce new issues if not carefully validated

TDD Tools and FrameworksHere is a list of some of the most popular and used TDD tools and frameworks:

• Jest: A leading testing framework in the JavaScript ecosystem. Jest is designed for modern web applications and requires minimal configuration. It comes with built-in support for mocking, snapshot testing, and test coverage. See how to write unit tests using Jest in Node.js.
• JUnit: A widely adopted framework for Java development. JUnit simplifies writing and running unit tests. It offers annotations for test methods and integrates seamlessly with IDEs and build tools like Maven and Gradle.
• pytest: A versatile and feature-packed testing framework for Python. pytest excels in supporting fixtures, parameterized tests, and plugins. Learn more in our guide on how to test Python applications with pytest.
• NUnit: A popular testing technology within the .NET ecosystem. NUnit provides a robust framework for writing and executing unit tests. It supports parameterized tests, setup/teardown methods, and a rich assertion library for TDD in .NET projects.

Test-Driven Development ExampleNow that you understand what TDD is, how it works, and the tools you can use to implement it, it is time to see a complete Test-Driven Development example.

Imagine you want to implement a feature to check whether a string is a palindrome. We will use Jest as the testing framework, but any other TDD tool will work.

This is the textual description of the functionality to implement:

“Check whether a given word or phrase is the same when read backward as it is forward (i.e., it is a palindrome).”

In the Red phase, you can write the following high-level test cases to verify both possible outcomes:

import { isPalindrome } from "utils.js"describe("isPalindrome", () => { it("should return true for a palindrome string", () => { // Arrange const input = "radar" const expectedOutput = true // Act const result = isPalindrome(input) // Assert expect(result).toBe(expectedOutput) }); it("should return false for a non-palindrome string", () => { // Arrange const input = "hello" const expectedOutput = false // Act const result = isPalindrome(input) // Assert expect(result).toBe(expectedOutput) })}) Keep in mind that the Red phase of TDD can involve writing multiple failing tests, especially when dealing with simple functionality that covers only a few possible scenarios. At this stage, running the tests will fail because the isPalindrome() function does not exist yet.

Note: These tests follow the AAA (Arrange, Act, Assert) pattern.

Next, in the Green phase, you have implement the isPalindrome() function to make the tests pass:

// utils.jsexport function isPalindrome(inputStr) { return inputStr === inputStr.split("").reverse().join("")} With this implementation, running the test case implemented earlier will now pass.

Finally, complete the Refactor phase to make the code capable of handling edge cases and more robust:

function isPalindrome(inputStr) { // to handle empty strings if (!inputStr) { return true } // normalize the input string to make the function // more robust const normalizedStr = inputStr.toLowerCase() return normalizedStr === normalizedStr.split("").reverse().join("")} Congratulations! You just used TDD to successfully implement the isPalindrome() function matching the desired functionality.

Get ready to dive into Behavior-Driven Development and compare it with TDD in this TDD vs BDD discussion!

What Is BDD (Behavior-Driven Development)?BDD, short for Behavior-Driven Development, is a collaborative software development approach that extends Test-Driven Development (TDD) and Acceptance Test-Driven Development (ATDD) by emphasizing communication and shared understanding among all team members.

BDD enhances TDD and ATDD with the following principles:

• Apply the “five whys” to each proposed user story, ensuring its purpose aligns with business outcomes. If you are not familiar with that term, a “user story” is an informal, natural language description that outlines the features, functionality, or requirements of a software application from the perspective of the end user.
• Think “from the outside in,” focusing on implementing only behaviors that directly contribute to these business outcomes.
• Use a simple, unified language for describing behaviors, accessible to domain experts, testers, and developers, to improve communication.
• Apply these practices throughout the software’s abstraction layers, paying attention to behavior distribution, making future changes easier and more cost-effective.

In simpler terms, Behavior-Driven Development defines the expected behavior of a software system using scenarios in natural language. This way, both technical and non-technical stakeholders can understand.

As explained by Dan North in his article “Introducing BDD” published in 2006, Behavior-Driven Development tries to address the challenges of TDD in writing effective tests. The ultimate goal is to bridge the gap between technical testing and business needs by framing tests as examples of desired behavior.

How to Perform BDDImplementing BDD revolves around writing behavior examples and then developing automated tests to verify that the software behaves as expected. This process usually involves three steps:

1. Discovery: Understand the requirements
2. Formulation: Define the acceptance criteria
3. Automation: Turn acceptance criteria into automated tests

Let’s explore those steps in detail!

Step#1: DiscoveryIn this phase, the team collaborates to turn business requirements into user stories. A user story provides a clear, concise description of the behavior to be implemented. For example:

“As a user, I want to be able to log in to my account so that I can access my dashboard.”

The goal is to ensure that everyone on the team understands the feature’s purpose and the value it delivers to users. After defining the user story, developers, testers, designers, and managers work together to also create initial acceptance criteria. These specify potential scenarios and edge cases to clarify expectations.

Step #2: Formulation PhaseDuring this phase, broad acceptance criteria established earlier are refined into specific scenarios. Initially, acceptance criteria may be vague or general. For example, the discovery phase might provide the following high-level acceptance criterion for the login user story:

“A user should be able to log in to their account with valid credentials.”

The purpose of this phase is to translate them into clear and concrete examples of the expected system behavior. That eliminates misunderstandings between business and technical teams.

To achieve the result, user stories are transformed into structured BDD scenarios—generally expressed through the “Given-When-Then” format:

• Given: Describe the initial state or pre-condition of the system before the behavior occurs.
• When: Describe the action or behavior being performed.
• Then: Describe the expected result or outcome of the behavior.

During the formulation phase, the simple criterion presented above can refined into a more detailed BDD scenario:

Scenario: Successful login Given the user is on the login page When the user enters valid credentials Then they should be redirected to their dashboard This scenario is written using Gherkin syntax, which is commonly supported by tools like Cucumber. This format ensures that all team members have a shared understanding of the feature’s behavior.

Step #3: Automation PhaseThe BDD scenario from the previous phase is automated through an acceptance test built using a BDD framework. Initially, the automated test will fail because the feature has not been implemented yet. However—as development progresses—and the code of the feature is implemented, the test will pass, validating that the feature works as intended.

Behavior-Driven Development ExampleConsider a Behavior-Driven Development (BDD) example to better understand how this methodology works.

Suppose the discovery phase lead to the following user story:

“As a user, I want to be able to sum two numbers”

That can be translated into a specific BDD scenario:

Scenario: Given two numbers, 5 and 3 When they are summed Then the result should be 8 Below is an acceptance test implemented with Jest, which also supports a BDD-like syntax:

import { sum } from "mathUtils.js"describe("Calculator", () => { it("should add two numbers correctly", () => { // Given const num1 = 5 const num2 = 3 // When const result = num1 + num2 // Then expect(result).toBe(8) })}) This automated test implements the Given-When-Then specifications. In particular, the describe() and it() methods are in line with in line with BDD principles as they allow you to structure tests in a human-readable format.

BDD ToolsBelow is a list of some of the most widely used BDD frameworks and tools:

• Cucumber: A BDD framework that supports writing tests in Gherkin syntax, enabling collaboration between business and technical teams. Find out more in our article on writing acceptance tests with Cucumber.
• Behave: A Python-based BDD framework that uses Gherkin syntax to define test cases, making it easy to validate application behavior against requirements.
• SpecFlow: A .NET BDD framework that supports Gherkin syntax and integrates well with testing tools like NUnit, MSTest, and xUnit for behavior validation.
• Behat: A PHP-based BDD framework, using Gherkin to describe application behavior in readable language, useful for functional testing.
• Ginkgo: A BDD-style testing framework for Go, supporting expressive tests written in a behavior-driven manner.

Can BDD and TDD Coexist?TL:DR: Yes, BDD and TDD can coexist and should be used together for maximum synergy.

To better understand the role of TDD and BDD in a software development cycle, take a look at the diagram below:

As shown in the image, TDD focuses on verifying that individual components function correctly through several iterations of the Red-Green-Refactor loop (the inner cycle on the right). On the other hand, BDD loop (the broader outer cycle) emphasizes collaboration and aligns the system’s behavior with user expectations.

When used together, BDD shapes the development process by guiding it to meet user stories and business goals, while TDD enforces that the code is well-tested and reliable. The two methodologies serve complementary roles and work in tandem. It is no surprise that many testing tools accommodate both the TDD methodology and the creation of automated tests through BDD-like syntax, just like Jest.

TDD addresses technical correctness, while BDD focuses on aligning code with user behavior. Together, they create a feedback loop that continuously improves both code quality and user satisfaction.

TDD vs BDD: Summary TableExplore the differences between Test-Driven Development and Behavior-Driven Development in the TDD vs BDD comparison table below:

| Aspect | Test-Driven Development | Behavior-Driven Development | | --- | --- | --- | | Acronym | TDD | BDD | | Creators | Kent Beck and others | Dan North and others | | Creation time | Late 1990s | Early 2000s | | Inspirations | Extreme Programming | Test-Driven Development (TDD) and Acceptance Test-Driven Development (ATDD) | | Goal | Verify code correctness at a granular level | Verify the system’s behavior aligns with user requirements and expectations | | Phases | – Red (Write failing test)– Green (Make test pass)– Refactor (Improve code) | – Discovery (Understand requirements)– Formulation (Write scenarios)– Automation (Write automated tests) | | Focus | Ensure that individual units of code work correctly | Ensure that the software behaves as expected from the user’s perspective | | Approach | Developer-centric | Collaborative | | Team | Developers, QA engineers, and other technical roles | Developers, QA engineers, testers, and non-technical stakeholders |

ConclusionIn this BDD vs TDD comparison guide, you learned about the differences between Behavior-Driven Development and Test-Driven Development. While BDD and TDD are often mistaken for one another, they represent two distinct methodologies—each with its unique characteristics and tools.

The post TDD vs. BDD: What’s the Difference? (Complete Comparison) appeared first on Semaphore.

View Details

Continuous Integration (CI) is a foundational practice in modern software development. It plays a crucial role in agile workflows as it promotes collaboration, faster release updates, and keeps software aligned with business needs. It encourages developers to integrate code into a shared repository frequently.

CI pipeline steps like builds and tests automatically verify each commit. This way, developers can catch errors early, and improve code quality.

In this article, we’ll go through ten essential rules to maximize your CI pipeline’s efficiency and reliability. These rules will help you optimize build speed, improve testing accuracy, and boost overall development processes. You can apply these best practices to new or existing pipelines.

Top 10 Rules to Maximize Your CI PipelineLet’s dive into each rule and see how it can help enhance your CI pipeline.

1. Use Version Control for a Shared Core RepositoryA reliable CI pipeline starts with a strong foundation: a version-controlled core repository. Version control (like Git) helps teams maintain a single source of truth, ensuring all code changes are traceable and easy to review. Version control enables smooth collaboration by allowing developers to work on branches, create pull requests, and conduct code reviews before merging changes.

In a CI setup, having all code in a central repository lets CI tools automatically pull the latest code for builds and tests. This centralization reduces conflicts, keeps everyone on the same page, and makes it easy to roll back changes if needed. Popular places to store your codebase are GitHub, GitLab, and Bitbucket.

1. Commit Code FrequentlyFrequent commits are essential for maintaining a healthy CI pipeline. Committing small, incremental changes allows developers to catch issues early and keep code reviews manageable. If a problem occurs, it is easier to roll back to a previous commit.

Large, infrequent commits increase the chances of conflicts and make it harder to pinpoint the cause of errors. Moreover, infrequent commits can lead to a significant risk of losing work. If your changes are only stored locally and your machine fails—whether due to hardware issues or accidental data loss—you could lose hours or even days of effort.

Therefore, developers working on feature branches should push their code at least once daily.

Pushing your code daily isn’t about having everything perfect—it’s about progress and teamwork. Even if your code isn’t finished, sharing it regularly lets the team see what you’re working on, sparks helpful discussions, and ensures you make the right decisions on time. Frequent commits foster transparency, collaboration, and trust within the team.

1. Automate BuildsAutomated builds are at the core of any CI pipeline. By setting up a CI tool to automatically build the application with each commit, you eliminate the need for manual builds and reduce the chance of human error. This is also a cost-efficient solution as you don’t need to hire experts solely to manage builds. Developers can focus on writing code instead of managing build processes. For example, platforms like SemaphoreCI offer a solution to set up your CI configuration.

Automated builds ensure new changes integrate smoothly with the existing code and verify that dependencies are installed correctly. Any broken build requires immediate attention. In a well-configured CI pipeline, the build step should fail if the build process encounters an issue. If someone pushes code that breaks the build, the CI pipeline will fail the build step. It will prevent the faulty code from being merged into the main branch, protecting the work of other team members who depend on a stable codebase. When a build is successful, it qualifies the application for further testing.

Build tools like Maven or Gradle handle tasks such as dependency management, compiling code, and packaging applications, making builds faster and more reliable. After a successful run, they typically generate artifacts, which are the outputs of the build process (e.g., JAR files, WAR files, or Docker images).

By integrating these tools into your CI pipeline, you ensure that builds are consistent across different environments.

1. Automate Testing at Every LevelTesting is vital in CI. The most common types of testing include unit, integration, and end-to-end testing.

2. Unit tests focus on testing individual functions or components in isolation. These tests should be fast to avoid bottlenecking the CI process. Each unit test should execute within milliseconds, ensuring it remains small, fast, and focused on testing a single functionality. While individual tests are quick, the full test suite might take several seconds to complete, depending on its size. This speed ensures rapid feedback during development.

Popular open-source tools for unit testing include JUnit (Java), Pytest (Python), and Jest (JavaScript).

• Integration tests validate that different modules or components work together correctly. They are slower than unit tests, typically taking a few seconds per test, because they require setting up dependencies like databases, APIs, or external services. Integration tests help ensure that communication between components is seamless.

Tools like TestContainers (for testing with real dependencies), Spring Boot Test (Java), and Mocha (JavaScript) are popular for integration testing.

• End-to-End (E2E) tests simulate user workflows by testing the entire application from start to finish. They ensure that all parts of the system work together as expected. E2E tests are significantly slower, typically running on the order of minutes per test, as they involve the complete system, including the user interface, backend services, and external integrations. While essential for catching system-wide issues, E2E tests should focus on critical paths to avoid bloating execution time in the CI pipeline.

Commonly used open-source tools for E2E testing include Cypress, Playwright, and Selenium.

Together, these tests form the foundation of the testing pyramid, a concept introduced by Mike Cohn, consisting of Unit Tests, Service Tests, and User Interface Tests.

Although it is useful, nowadays experts consider it too simplistic and potentially misleading. However, you can still implement some valid key points to design more efficient test suites:

• Focus on Test Granularity: Write a higher number of small, fast unit tests, some coarse-grained integration tests, and a few high-level end-to-end tests. This ensures a balanced, maintainable, and efficient test suite.
• Pyramid Shape: Stick to the pyramid’s structure to avoid creating the ATICC antipattern (Automated Testing Ice Cream Cone). It occurs when testing efforts focus too heavily on manual and high-level tests (e.g., UI or end-to-end), neglecting automated unit tests. This imbalance slows development, increases the risk of undetected bugs, and makes it harder to respond quickly in agile environments. To avoid this, prioritize automating unit tests while maintaining a balanced testing pyramid.

In a CI pipeline, unit tests act as the first line of defense, catching issues early before the code progresses further down the pipeline. If unit tests fail, it’s pointless to proceed with higher-level tests like integration or end-to-end tests. For example, Semaphore offers a fail-fast feature that can stop all remaining jobs when a job fails.

Another useful testing approach is static code analysis. For example, consider integrating lint into your CI. It automatically checks the source code for syntax errors, stylistic issues, and potential bugs. It helps ensure the code adheres to predefined coding standards and best practices, improving code quality and readability. You can use linting tools like ESLint for JavaScript, Pylint for Python, or Checkstyle for Java to automate this process.

You can also enable autolinting directly in your IDE, such as IntelliJ IDEA and VS Code. You can configure plugins like ESLint to automatically fix issues upon saving by adjusting the IDE’s settings. This ensures code quality as you type.

1. Implement Smart Triggers and Branch-Specific PipelinesEfficient CI pipelines use smart triggers and branch-specific pipelines to optimize resource usage. Instead of executing the entire pipeline on every branch with each commit, you can set up path-based or branch-based triggers to control what jobs to run. For example, commits to the main branch might trigger full builds, while feature branches could run a smaller set of tests to save resources.

As mentioned earlier, unit tests are the foundation of any testing process and should be prioritized, while integration tests require more time and resources. Semaphore’s conditions provide the flexibility to define triggers based on regular expressions.

1. Automate Code and Security ChecksIncorporating automated code quality and security checks into your CI pipeline helps ensure that your code remains maintainable and secure. Tools like SonarQube can automatically analyze your codebase for quality issues, such as bugs, code smells, and technical debt.

Additionally, security scanning tools like Trivy or Snyk can catch vulnerabilities in both your code and any dependencies you’re using. These tools automatically and continuously scan for known vulnerabilities, alert the team with findings and can even block branches with security flaws from being merged into the main branch.

1. Enable Notifications and Feedback LoopsInstant feedback is crucial for maintaining a smooth development process. By enabling notifications for build or test failures, teams can quickly address issues as they arise. For example, sending Slack or email alerts helps ensure that developers know immediately when something goes wrong, reducing downtime and frustration.

In addition to notifications, automated reporting tools like SonarQube can provide detailed insights into code quality, test coverage, and other metrics.

Semaphore also offers a test reports dashboard. Semaphore’s Test Reports highlight critical failures and provide detailed insights into your test suite. You can filter tests and sort them by failure order or slowness, making it easier to prioritize the most pressing issues. When a critical failure occurs, teams can treat it as a “stop the line” event. Everyone drops other tasks to collaborate and resolve the problem immediately. This approach keeps the CI pipeline healthy and ensures reliable software delivery.

These feedback loops help teams improve code quality over time and prevent unexpected problems.

1. Mirror Production Environments for TestingTo ensure reliable and consistent testing, it’s essential to mirror production environments as closely as possible. Using tools like Docker to containerize applications helps create isolated, reproducible environments for testing.

Additionally, tools like TestContainers can be used to spin up necessary dependencies (e.g., databases, message brokers) in containers during tests. Running tests in containers or using TestContainers helps verify that the code behaves as expected in real-world conditions.

If you deploy your application to Kubernetes in production, it’s important to mirror that environment in CI for integration tests. Tools like Minikube or Kind allow you to run local Kubernetes clusters within your CI pipeline to test containerized applications in an environment that mimics production.

However, be careful to avoid heavy end-to-end tests, resulting in slow feedback and inefficiency. Instead, reserve production-like environments for the most critical integration test paths and ensure earlier pipeline stages cover comprehensive unit and functional testing.

1. Monitor and Measure CI PerformanceContinuous monitoring and measuring the performance of your CI pipeline is crucial for identifying bottlenecks and optimizing the workflow. Regularly monitoring CI performance allows teams to proactively address issues, rather than waiting for problems to arise.

Using tools like Semaphore CI Project Insights, you can track key metrics such as build times, test durations, and success rates. These insights help you identify areas for improvement, such as slow-running tests or inefficient build steps.

Semaphore also allows you to detect flaky tests. These are tests that are unreliable and fail randomly for no obvious reason. The flaky test detection works automatically once you set up test reports.

You can also conduct developer surveys to find areas for improvement. For example, you can identify pain points in the workflow, such as excessive waiting times, complex setup processes, lack of visibility, etc.

A good rule of thumb for optimizing CI pipelines is to keep build stages under 10 minutes to avoid developer frustration and provide time for breaks. Similarly, aim to keep test execution times under 5 minutes for fast feedback.

By addressing these issues, you’ll improve the overall efficiency and developer experience.

1. Optimize Build and Test PerformanceEfficiency in your CI pipeline is essential for saving time and resources.

One way to optimize build and test performance is by implementing dependency caching. Caching dependencies that don’t change frequently can significantly speed up builds, as your CI system doesn’t have to download or compile them every time. For example, Semaphore provides a caching mechanism to help speed up your builds.

Additionally, parallelizing tests allows you to run multiple tests simultaneously, reducing overall test execution time. You can use techniques like Semaphore’s job parallelism to speed up your test suites.

For complex systems, mocking external services or environments that aren’t essential to the test can further speed up execution by eliminating unnecessary dependencies. Tools like WireMock, MockServer, and Moco can simulate external APIs or services, reducing the need for actual network calls during testing. This allows you to isolate tests and ensure they run faster and more reliably.

ConclusionIn this article, you’ve learned the top 10 best practices to get the most out of your CI pipeline. To summarize, here’s a visual representation of a complete CI pipeline, incorporating the best practices we’ve discussed:

Following these rules will help improve the overall development cycle, reduce costs, and accelerate software delivery.

Thank you for reading!

The post Top 10 Rules of Continuous Integration appeared first on Semaphore.

View Details

Web applications are increasingly transforming browsers into the primary interface for our digital lives. As we rely more on web applications for everything from work to entertainment, the responsiveness of these applications becomes paramount. Rich Harris, a former journalist and the mind behind Svelte, shares his insights into web development technologies and trends.

Edited transcriptionA journalist by trade, Rich Harris recognized the potential of web technologies to enable “new forms of interesting and exciting journalism that involve visualizing large datasets.” Curious about how these technologies could help him do a better job, he began learning JavaScript and eventually open-source software development: “I began building tools for myself and then sort of sharing them on GitHub, discovering that other people had the same needs,” he reflects. Ultimately, Rich kept diving deeper into software development and created Svelte and its companion framework, SvelteKit, and transitioned out of journalism when he secured sponsorship from Vercel for his project.

Svelte is a JavaScript framework for building user interfaces. Unlike traditional frameworks like React and Vue, Svelte compiles components into highly optimized JavaScript code, resulting in faster performance from the client and smaller bundle sizes. Rich describes Svelte as “a way of building interactive websites with a lot more reliability and a lot less pain than if you were manually wiring up all of the DOM manipulation code yourself.”

Building the frameworkThe most defining philosophical difference between Svelte and other frameworks is the prominence of the compiler. As Rich puts it, “It doesn’t make sense to do a lot of work in the browser when every single user has to do that work and you have to ship the code that does that work when you could just do that at compile time.”

The compiler allows writing code “in this sort of declarative form that very closely resembles HTML, but with interactivity added on top of it,” Rich explains. Hence, unlike JavaScript-centric frameworks, Svelte allows developers to use JS, HTML, and CSS in a more integrated manner. “When you build a component in Svelte, you can include styles,” Rich notes. This scoped CSS approach ensures that styles defined within a component are isolated to that component. “It makes sense if that CSS belongs with the component in such a way that the framework can keep the styles defined with the components scoped to that component, which is something that Svelte does.”

In this regard, Rich believes other frameworks often “lean more in the direction of letting the community solve” issues. While he acknowledges that this can produce excellent solutions—since “people can really focus very intensely on one specific problem”—it also places a burden on developers. “Everyone who uses the framework needs to spend time thinking about which of the many solutions on display they’re going to use, and which is right for their application,” he says. In contrast, Svelte puts developer experience first: Users “want to be able to start writing code and be productive, and have the answer ready to the question, ‘How do I do this thing?’ before they start working.”

Many of Svelte’s features were shaped by this principle, including built-in support for animations and transitions as first-class primitives. “Whereas in other frameworks, you might need to research which library to use and then figure out how to integrate it, Svelte just supports these features out of the box,” says Rich. Likewise, Svelte’s implementation of signals introduced a more flexible way to manage state and dependencies. Signals are a mechanism for building reactive systems that automatically update the UI whenever their values change: “We’re able to be extremely surgical about which parts of the page update in response to a given change,” he says.

Finding the balance between server and client workloadWhile Svelte has been optimized to prioritize build-time tasks, there are inherent limits to what can be achieved with it. As Rich confesses, “When you’re building a large enough application, you will find that there are things that you just cannot know at build time; you need to have the information of the dependency graph available at runtime, and that just wasn’t there.”

Shifting workloads between the browser and the server is not a settled matter; in fact, it has become a recurring trend in web development, fed by hype cycles. “We shouldn’t be surprised that we’re in the middle of another one of these pendulum swings right now,” says Rich. To his understanding, these cycles often overcorrect previous trends. For example, during the 2010s, developers increasingly shifted workloads to the browser following the Ajax revolution sparked by Gmail, which led to the proliferation of large JavaScript bundles that often overloaded the browser. In turn, over the past few years, there’s been a shift back to server-side rendering, a trend of which Svelte is a part of.

Withal, Rich advocates for a balanced approach. “Code should run where it’s needed. If you have a supercomputer in your pocket and the ability to render a high-fidelity interactive user experience using code that runs on that device, then do it there.” Web applications can learn from native apps, which operate offline; excessive reliance on servers undermines offline functionality—a feature many users value.

To balance the workload, Svelte has SvelteKit, a framework built on top of Svelte for creating modern web applications. For enabling offline functionality, SvelteKit supports Progressive Web App (PWA) features. For example, SvelteKit can be configured to use service workers—scripts that run in the background to cache essential assets like HTML, CSS, and JavaScript files. When users revisit the application, these cached assets allow the app to load instantly, even without a server connection. SvelteKit also supports pre-rendering pages at build time, generating static HTML for specific routes or pages. These pre-rendered pages can be cached on the user’s device, ensuring they remain accessible offline. Additionally, SvelteKit facilitates API requests and client-side data fetching. With caching strategies, developers can store API responses and other data offline using technologies like service workers or IndexedDB.

The bottom lineTo get involved with Svelte, Rich encourages developers to create component libraries, which are essential building blocks for applications. Visit github.com/sveltejs to learn more. Rich also invites enthusiasts to join the Svelte community on Discord at svelte.dev/chat.

You can follow Rich in X at @rich_harris.

The post Rich Harris on the Future —and Past— of Web Development appeared first on Semaphore.

View Details

Security is a critical concern in modern software development, especially when dealing with open-source dependencies. Vulnerabilities in these libraries can introduce severe risks to your application, exposing it to potential attacks. That’s why it is essential to ensure that your codebase remains secure.

Snyk, a powerful open-source security tool, helps you to identify and fix vulnerabilities in your codebase. When you combine it with Semaphore CI’s CI/CD pipeline, you can automate these checks and ensure that your software stays secure at every step of the development process.

In this article, you will learn:

• How Snyk works, with a focus on Snyk Open Source.
• How to integrate Snyk with Semaphore CI for automated vulnerability scanning.
• Best practices for securing your CI/CD pipeline.

Let’s get started!

Snyk OverviewWhat is Snyk?

Snyk is a comprehensive platform designed to secure every aspect of your application stack, from code to infrastructure. It provides a suite of tools to identify and remediate vulnerabilities in your codebase, including:

• Snyk Code: Detects vulnerabilities in your source code.
• Snyk Container: Secures container images against vulnerabilities.
• Snyk Infrastructure as Code (IaC): Fixes misconfigurations in IaC tools like Terraform and Kubernetes.
• Snyk Open Source: Scans and resolves vulnerabilities in your open-source dependencies.

One of Snyk’s standout features is its deep integration with CI/CD pipelines. It allows you to automatically run security scans.

Snyk offers a free tier, making it easy for individual developers and small teams to get started.

In this article, we’ll focus on Snyk Open Source. It automatically identifies and fixes vulnerabilities in open-source libraries and dependencies. It scans your project’s libraries, flags security risks, and offers advice to remediate the vulnerabilities.

Snyk Open Source scans your project’s dependencies for vulnerabilities using its continuously updated vulnerability database. This database pulls from various sources, including the National Vulnerability Database (NVD), security advisories, GitHub issues, and Snyk’s in-house security research team.

How to Use Snyk?

You can utilize Snyk in several ways:

• Web UI: Manage, configure, and fix issues through Snyk’s web interface.
• CLI: Integrate Snyk into your local machine or CI/CD pipelines (which you will see in this article).
• IDEs: Use Snyk within your development environment.
• API: Programmatically integrate Snyk into your workflows (available only for Enterprise plans).

Setting Up SnykFollow these steps to get started with Snyk.

1. Create a free Snyk Account by visiting the Snyk website.
2. To run security checks locally, install the Snyk CLI:

curl --compressed https://downloads.snyk.io/cli/stable/snyk-linux -o snykchmod +x ./snyksudo mv ./snyk /usr/local/bin/ Check the installation guide for more installation options.

1. After installation, authenticate Snyk CLI with your account by running:

snyk auth This command will open a web browser where you can log in to your Snyk account. Finally, you should see the following messages in your Terminal:

Opening in existing browser session.Your account has been authenticated. 4. Navigate to your project’s directory and run the following command to scan for vulnerabilities:

snyk test Snyk will analyze your project and provide a report of any existing vulnerabilities. For example:

Tested 5 dependencies for known issues, found 5 issues, 5 vulnerable paths.Issues to fix by upgrading: Upgrade ch.qos.logback:logback-classic@1.2.3 to ch.qos.logback:logback-classic@1.2.13 to fix ✗ Insufficient Hostname Verification [Medium Severity][https://security.snyk.io/vuln/SNYK-JAVA-CHQOSLOGBACK-1726923] in ch.qos.logback:logback-core@1.2.3 introduced by ch.qos.logback:logback-classic@1.2.3 > ch.qos.logback:logback-core@1.2.3 ✗ Denial of Service (DoS) [High Severity][https://security.snyk.io/vuln/SNYK-JAVA-CHQOSLOGBACK-6094943] in ch.qos.logback:logback-core@1.2.3 introduced by ch.qos.logback:logback-classic@1.2.3 > ch.qos.logback:logback-core@1.2.3 ✗ Uncontrolled Resource Consumption ('Resource Exhaustion') [High Severity][https://security.snyk.io/vuln/SNYK-JAVA-CHQOSLOGBACK-6097493] in ch.qos.logback:logback-core@1.2.3 introduced by ch.qos.logback:logback-classic@1.2.3 > ch.qos.logback:logback-core@1.2.3 ✗ Denial of Service (DoS) [High Severity][https://security.snyk.io/vuln/SNYK-JAVA-CHQOSLOGBACK-6094942] in ch.qos.logback:logback-classic@1.2.3 introduced by ch.qos.logback:logback-classic@1.2.3 ✗ Uncontrolled Resource Consumption ('Resource Exhaustion') [High Severity][https://security.snyk.io/vuln/SNYK-JAVA-CHQOSLOGBACK-6097492] in ch.qos.logback:logback-classic@1.2.3 introduced by ch.qos.logback:logback-classic@1.2.3 Optionally, you can link your repository (like GitHub or Bitbucket) to allow Snyk to scan your project for vulnerabilities. However, we will skip this step as our focus will be on using Snyk within the CI/CD pipeline process.

Integrating Snyk with Semaphore CINow that you know how Snyk works, let’s integrate it into the Semaphore CI pipeline.

1. First, you need a free Semaphore account. Visit the Semaphore website and choose either GitHub or Bitbucket for registration. In this tutorial, I will use GitHub.

2. Add a new project by clicking the “+Create New” button.

I’ve created a simple Maven project with minimal source code and a single JUnit test. I will connect my repository with Semaphore:

1. Click on “Continue to workflow setup”. Semaphore will automatically initialize the project. You can add more people to it if you wish.
2. Next, we need to create the workflow for the pipeline. You can configure the Semaphore pipeline via the UI or manually create the semaphore.yml file in your project. I will show you the first approach.

3. Select the “Maven” template from the list. Click on “Customize” to override the default configuration:

4. Change the default machine type to e1-standard-2. The e1 series machines are appropriate for less compute and memory-intensive jobs.

5. Edit the Build block to combine the build and test steps into a single block since they share the same initial commands. Instead of duplicating these commands for each job, we can use the Prologue section to streamline the process. Add the shared commands to the Prologue, ensuring they run once before the jobs.

The prologue commands:

checkoutsem-version java 17 This command checks out the project’s code. The sem-version utility on Linux-based virtual machines allows you to set or switch the version of a programming language.

The build job command:

mvn -B package --file pom.xml This command compiles the project’s source code and packages it using Maven, creating a distributable artifact.

The test job command:

mvn test This command executes the project’s unit tests using Maven.

• Click “+ Add Block” to add a job for the security scan:

Security scan commands:

checkoutcurl --compressed https://downloads.snyk.io/cli/stable/snyk-linux -o snykchmod +x ./snyksudo mv ./snyk /usr/local/bin/SNYK_TOKEN=$SNYK_TOKEN snyk test These commands install the Snyk CLI tool, make it executable, place it in a system directory, and initiate a vulnerability test using an authentication token. We reference the $SNYK_TOKEN variable, which serves for authentication with Snyk. As a best practice, this token is stored as a secret to avoid exposing sensitive information. To create the secret, expand the “Secrets” section and open “Manage Secrets” in a new tab.

Select “Set up global secret” and name it “SNYK_TOKEN“. Input the token value, which you can retrieve from your Snyk account’s General Settings.

Click on “Save Secret”.

1. Go back to the workflow. The new secret name should appear under the Secrets section. Select it:

2. Click the “Run the workflow” button, and Semaphore will automatically start the pipeline. Additionally, it will create a new branch called “setup-semaphore” in your GitHub repository.

If you prefer the YAML creation approach, check out the link to my GitHub repository, which contains the content of the semaphore.yml file. Keep in mind that you still have to create the secret manually.

Testing the WorkflowThe pipeline should pass because we don’t have any vulnerabilities yet:

Let’s add an open-source library with known vulnerabilities to test the Security Scan job. Add this dependency to the pom.xml:

<dependency> <groupId>commons-beanutils</groupId> <artifactId>commons-beanutils</artifactId> <version>1.9.2</version> <!-- Known vulnerability in this version --></dependency> The pipeline should fail this time:

Best Practices for Secure CI/CD PipelinesTo maintain a secure CI/CD pipeline, consider these best practices for integrating security into your development workflow:

• Run security scans on every pull request: Automate security scans to ensure every change is evaluated for vulnerabilities before merging.
• Enforce policies against vulnerable code: Implement policies that prevent vulnerable code from being deployed.
• Regularly update dependencies: Keep your dependencies up to date to minimize exposure to known vulnerabilities.
• Store secrets as environment variables: Use environment variables or secret management tools to securely store sensitive information, like API tokens, and avoid hardcoding them in your pipeline configuration.
• Break the build for critical issues: Set up your pipeline to fail if high-priority vulnerabilities are found, ensuring they are addressed before deployment.

ConclusionIncorporating Snyk into your Semaphore CI pipeline is a powerful step towards enhancing the security of your applications.

Throughout this article, you learned:

• How Snyk, particularly Snyk Open Source, operates and what its key features are.
• The step-by-step process for integrating Snyk with Semaphore CI for automated vulnerability scanning.
• Best practices for securing your CI/CD pipeline, such as running security scans on every pull request and enforcing deployment policies against vulnerable code.

Thank you for reading!

The post Securing Your CI/CD Pipeline with Snyk Open Source and Semaphore appeared first on Semaphore.

View Details

You’re getting started with a new tool. You need to find your way around the documentation quickly. You’re not sure where to start reading. But imagine if someone could tell you which part of the documentation you should start reading first. So let’s build an AI that does that. Even if the documentation is partially written.

We will be building a bot that answers questions about the Semaphore documentation. You will learn how to build a RAG chatbot. And you will also build a tool that will help you get started with Semaphore.

But this will not be an ordinary RAG chatbot. It will also be useful while the documentation is still incomplete. So you will also learn a technique for working with sparse data.

What we are actually buildingWe are building a Python chatbot that will answer questions based on the new Semaphore documentation on GitHub. The bot will run in the terminal. It will answer questions about the new Sempahore documentation. It will provide a short answer and then direct you to the source document where you can read more.

These types of chat with documents systems are called Retrieval-Augmented Generation (RAG). When the user asks a question, the retriever searches for the relevant information. The retrieved information is inserted into the LLM’s prompt. And the LLM is instructed to generate the answer to the user’s question. The LLM’s answer must be based solely on the information provided in the prompt.

The LLM has a context window of a fixed size. The size of the input context window limits how much text we can include in the prompt. The size of the output context limits how much text the LLM can output. Therefore, we only augment the prompt with the most relevant chunks of documents. Because in most cases, we can’t fit all of our source documents into a single prompt.

In production, the source documents are stored in the system’s search database. In this tutorial, the documents will be stored in memory. Because we are building a first iteration quickly.

The code in this tutorial runs on Python 3.10.9, and the uses the following packages from pip.

langchain==0.3.0langchain-openai==0.2.0langchain-community==0.3.0faiss-cpu==1.8.0.post1 The LLM that we are using is gpt-4o-mini. But you could swap it for any LLM that you prefer to use.

Is the data usable? This is the greatest challengeThe greatest challenge is that the documentation is still being written. At the time of writing, the new documentation is very sparse. Some documents consist of three letters, “WIP”. Some documents are mostly example code. Some documents are mostly a diagram or two. You can imagine a presenter at a data conference bleating “Garbage-In Garbage-Out!” But it’s my job to deliver a solution despite the sparse data. So here’s the solution that I came up with.

We can often guess the purpose of a document before it has been completed. We look for clues. For example, a file called “SECURITY.md” will probably be about security. Even if the contents of that file are just three letters, “WIP”.

Let’s imagine another example. A file that contains a single YAML template, and no text to describe the template. This file is probably related to config.

We will be using the LLM to augment our documents with information that is implied by their path, file name, and content so far. Some documents have a lot of text already. Others have a few letters. To make every “document” is roughly the same size, we will use the LLM to generate summaries of those documents. Our RAG chatbot will only use the document summaries and the augmented context. The bot will explicitly direct the user to the source documents in each answer.

At the time of writing, the document preprocessing step takes approximately five minutes. And it costs less than $0.10 in OpenAI fees.

Anthropic have recently published an article that also describes a technique of augmenting data for RAG. In a typical RAG use case, the documents are split into chunks. When the user asks a question, the retriever retrieves only the most relevant chunks. Usually from several different source documents. In Anthropic’s Contextual Retrieval, they generate additional context for each chunk of text. This can help the retriever understand what the source document was talking about. In our case, we are not chunking our “documents”. For each source document, we are retrieving the whole summary plus augmented context.

For example, here is a “summary plus augmented context” for docs/getting-started/guided-tour.md.

The path 'docs/getting-started/guided-tour.md' suggests that this document serves as an introductory guide for new users of SemaphoreCI, helping them navigate the initial steps of setting up and using the tool effectively.This document provides a guided tour for new users of SemaphoreCI. It outlines essential tasks such as signing up, creating an organization, setting up a first project, and understanding Continuous Integration and Continuous Delivery processes. The document aims to help users get started with SemaphoreCI quickly and effectively, ensuring they can leverage the tool for their CI/CD workflows.The 'Guided Tour' document is a 20-minute tutorial designed to help new users of SemaphoreCI get acquainted with the platform. It emphasizes that no credit card is required to try out Semaphore, making it accessible for users to explore its features without any commitment. The tutorial covers four key learning objectives: first, it guides users through the sign-up process and organization creation; second, it introduces a simple 'Hello, World!' project to demonstrate the basic functionalities of CI; third, it provides a real-world example of Continuous Integration, including build and test steps; and finally, it explains the Continuous Delivery process, which includes releasing and deploying apps. Overall, this document serves as a foundational resource within the broader documentation, aimed at onboarding users efficiently and encouraging them to leverage SemaphoreCI for their development workflows. Another example is SECURITY.md. At the time of creating this tutorial, the file is a work in progress literally containing just three letters: “WIP”. On their own, these three letters would be useless to the retriever. Here is a “summary plus augmented context” for SECURITY.md:

The path 'SECURITY.md' suggests that this document is focused on security practices, guidelines, or considerations relevant to the Semaphore CI/CD tool. It likely outlines how users can secure their CI/CD processes, manage sensitive data, and comply with security standards.This document is intended to provide insights into the security protocols and best practices associated with using Semaphore CI/CD. Security is a critical aspect of CI/CD tools as they handle code, configurations, and potentially sensitive data. This document will help users understand how to implement security measures within their Semaphore workflows, contributing to overall operational security.The document labeled 'Security' is currently marked as 'WIP' (Work In Progress), indicating that it is not yet complete. While no specific content is provided, the intention behind the document is to outline security practices and considerations for using SemaphoreCI. Once fully developed, it is expected to cover topics such as securing deployment pipelines, managing access controls, ensuring the integrity of build artifacts, and maintaining compliance with security regulations. This document will be part of the broader SemaphoreCI documentation suite, which addresses various functionalities such as Continuous Integration, Pipelines, and Automation, among others. It will be particularly useful for developers and operations teams looking to implement secure CI/CD practices. Our prompt for summarizing and augmenting the documentsHere’s the prompt template that we will be using to summarize and augment the source documents. Notice that I wrapped it in a Python multiline string. A prompt template is different to a finished prompt. A prompt template has space for us to insert our data. The strings {path} and {doc} will be replaced by data. They will hold the path to a source document and the text of the source document.

This Python string will live in a file called prompts.py. prompts.py contains three python strings. You will see the other two strings later in this tutorial.

```

This is the prompt for summarizing and augmenting the docssummarize_and_augment_prompt = """The document in the "document" XML tags is part of the documentation for a CI/CD tool called SemaphoreCI.The path in the "path_to_document" XML tags is path to that particular document. Pay attention to the file and folder names because they suggest the purpose of the document.Semaphore is a CI/CD solution to streamline developer workflows.Semaphore features Continuous Integration and Pipelines, Deployments and Automation, Metrics and Observability, Security and Compliance, a Developer Toolkit, Test Reports, Monorepos, and Self-hosted build agents.Semaphore works in the cloud, on-premises, and in the hybrid cloud.{path}{doc}The document in the "document" XML tags is part of the documentation for a CI/CD tool called Semaphore.The path in the "path_to_document" XML tags is path to that particular document. Pay attention to the file and folder names because they suggest the purpose of the document.Summarize the document. Add context to the document as well. You will need to explain what task it is useful for. And where this document fits into the broader documentation.Use the following JSON template.{{ "what_path_suggests": Look at the path to the document. What does the path suggest about this document's purpose? "what_is_the_document_about": What is this document about? How does it help us use SempahoreCI? "summarize_document": Summarize the document. If the document has a lot of text, then write many paragraphs. If the document has code or a template, describe what you think tha template is for. What language is the code or template in?}}"""

`` Building the retrieverPreprocessing the documentsThe new Semaphore documents are stored in a git repo. We will first clone the repo to our local disk. Then we will use Pythonos.path.walk()` to iterate through each document. For each document, we will call the LLM to summarize and augment it. We will also generate an embedding for the combination of the summary and the augmented context.

The code below highlights the important details. The complete code file will be presented later in this article.

How to iterate through each document in the repo import os#1. Setup# Define the directory containing the markdown filesdocs_directory = "../semaphore/docs/"#3.0 Loop through and summarize the markdown files.for root, dirs, files in os.walk(docs_directory): #3.1 We loop through the markdown files only for filename in [ fi for fi in files if fi.endswith(".md") ]: file_path = os.path.join(root, filename) #3.2 Read the contents of the markdown file with open(file_path, "r") as file: content = file.read() #...Then we call the LLM for each document in a loop here. How to use the LLM to summarize and augment each documentWe are using the Lanchain library. Note that we are not using any of Langchain’s output parsers. We are using OpenAI’s JSON mode to get a structured output. Our chain returns a dictionary object. To enable JSON mode, we use model = ChatOpenAI(model=model_to_use).with_structured_output(None, method="json_mode").

We are also creating metadata for each document. We are creating a field called source. The source of a document is the url of its location on github. The chatbot will display the full URL when answering the user’s questions. The user will be able to click on the URL and view the document on github. The code below shows how it’s done.

import prompts #This is prompts.py, which I mentioned above.from langchain_openai import ChatOpenAIfrom langchain_core.prompts import ChatPromptTemplate#1. Setup# Define the directory containing the markdown filesclickable_url_prefix = "https://github.com/semaphoreci/semaphore/tree/main/docs/"docs_directory = "../semaphore/docs/"model_to_use = "gpt-4o-mini"# Check if output_dir is missing. mkdir if missing.if not os.path.exists(output_dir): os.makedirs(output_dir)summaries = []summary_metadata = []#2.0 Setting the Langchain objects##2.1 Setting up a Langchain "model" objectmodel = ChatOpenAI(model=model_to_use).with_structured_output(None, method="json_mode")##2.2 Setting up our prompt templateour_prompt_template = ChatPromptTemplate.from_messages( [ ("user", prompts.summarize_and_augment_prompt)])#3.0 Loop through and summarize the markdown files.for root, dirs, files in os.walk(docs_directory): #3.1 We loop through the markdown files only for filename in [ fi for fi in files if fi.endswith(".md") ]: file_path = os.path.join(root, filename) #3.2 Read the contents of the markdown file with open(file_path, "r") as file: content = file.read() #3.3 Use the LLM to summarize the markdown file our_path = file_path.replace(docs_directory,"") clickable_url = clickable_url_prefix + our_path chain = our_prompt_template | model json_formatted = chain.invoke({"path": our_path, "doc": content}) #3.4 Append to the summary and path to the lists the_summary = json_formatted["what_path_suggests"] the_summary += "\n" the_summary += json_formatted["what_is_the_document_about"] the_summary += "\n" the_summary += json_formatted["summarize_document"] the_summary += "\n" summaries.append(the_summary) summary_metadata.append({"source": clickable_url}) How to compute the embeddingsOur retriever searches based on embeddings rather than based on text. Next we need to compute embeddings based on the text that we have just generated. We will compute the “document” embeddings once and then save them to disk.

When the user asks our chatbot a question, we compute the embedding for the question. We then compare our embedded question with the embeddings of our “documents”. We retrieve the documents with the most similar embeddings. The exact implementation is called FAIS and we’ll be using a library for it.

At the end of the code snippet below, we set up a dictionary object that holds the text to be retrieved, the embeddings, and the metadata. In this case, the metadata is the path to the source document. We use the pickle library to save this dictionary object to disk.

import picklefrom langchain_openai import OpenAIEmbeddings#1. Setup# Define the directory containing the markdown filesclickable_url_prefix = "https://github.com/semaphoreci/semaphore/tree/main/docs/"docs_directory = "../semaphore/docs/"output_dir = "summarized_docs/"model_to_use = "gpt-4o-mini"output_pickle_file_name = "texts_and_embeddings.pkl"output_pickle_file_path = os.path.join(output_dir, output_pickle_file_name)#...Document looping/generation code has been omitted for clarity.#4.0 Generate the embeddingsprint("Embedding summaries...")summary_embeddings = embeddings.embed_documents(summaries)#5.0 zip everything together and save to disktext_embedding_metadata_triplets = {"embeddings":zip(summaries, summary_embeddings), "meta": summary_metadata}with open(output_pickle_file_path, "wb") as p_file: pickle.dump(text_embedding_metadata_triplets, p_file) The complete preprocessing fileWe store the code to preprocess the documents in a file called summarize_docs.py.

```

This file summarizes the documents, creates embeddings, and saves the data to a pickle file.import osimport pickleimport promptsfrom langchain_openai import ChatOpenAIfrom langchain_openai import OpenAIEmbeddingsfrom langchain_core.prompts import ChatPromptTemplate#1. Setup# Define the directory containing the markdown filesclickable_url_prefix = "https://github.com/semaphoreci/semaphore/tree/main/docs/"docs_directory = "../semaphore/docs/"output_dir = "summarized_docs/"model_to_use = "gpt-4o-mini"output_pickle_file_name = "texts_and_embeddings.pkl"output_pickle_file_path = os.path.join(output_dir, output_pickle_file_name)# Check if output_dir is missing. mkdir if missing.if not os.path.exists(output_dir): os.makedirs(output_dir)summaries = []summary_metadata = []# Optional: Prompt the user for the API key#import getpass#os.environ["OPENAI_API_KEY"] = getpass.getpass()#2.0 Setting the Langchain objects##2.1 Setting up a Langchain "model" objectmodel = ChatOpenAI(model=model_to_use).with_structured_output(None, method="json_mode")##2.2 Setting up our prompt templateour_prompt_template = ChatPromptTemplate.from_messages( [ ("user", prompts.summarize_and_augment_prompt)])#2.3 Set up the embeddings modelembeddings = OpenAIEmbeddings()#3.0 Loop through and summarize the markdown files.for root, dirs, files in os.walk(docs_directory): #3.1 We loop through the markdown files only for filename in [ fi for fi in files if fi.endswith(".md") ]: file_path = os.path.join(root, filename) #3.2 Read the contents of the markdown file with open(file_path, "r") as file: content = file.read() #3.3 Use the LLM to summarize the markdown file our_path = file_path.replace(docs_directory,"") clickable_url = clickable_url_prefix + our_path chain = our_prompt_template | model json_formatted = chain.invoke({"path": our_path, "doc": content}) #3.4 Append to the summary and path to the lists the_summary = json_formatted["what_path_suggests"] the_summary += "\n" the_summary += json_formatted["what_is_the_document_about"] the_summary += "\n" the_summary += json_formatted["summarize_document"] the_summary += "\n" summaries.append(the_summary) summary_metadata.append({"source": clickable_url}) #3.5 Optional: Write the sumarised file to disk s_file_name = os.path.join(output_dir, our_path.replace("/","_")) with open(s_file_name, "w") as s_file: print(s_file_name) path_spec_text = "Path to file: {0} \n\n".format(our_path) s_file.write(the_summary)#4.0 Generate the embeddingsprint("Embedding summaries...")summary_embeddings = embeddings.embed_documents(summaries)#5.0 zip everything together and save to disktext_embedding_metadata_triplets = {"embeddings":zip(summaries, summary_embeddings), "meta": summary_metadata}with open(output_pickle_file_path, "wb") as p_file: pickle.dump(text_embedding_metadata_triplets, p_file)

``` How to retrieve the relevant documents at query timeWe’re using FAIS as our retriever. I chose FAIS for this tutorial because it runs in memory. We don’t need to worry installing any databases. In production, you might decide to outsource the retriever to a cloud vendor’s search product.

The code below is an extract from a larger file that is called rag_bot.py. The complete file will be presented later in this article.

We read the pickled data from disk. Then we instantiate a retriever object. To test our retriever, we could search our documents with retriever.invoke(). For example, retriever.invoke("What do I need to keep in mind for security?")

import osimport picklefrom langchain_openai import OpenAIEmbeddingsfrom langchain_community.vectorstores import FAISS#1. Setupdocs_dir = "summarized_docs/"docs_pickle_file_name = "texts_and_embeddings.pkl"docs_pickle_file_path = os.path.join(docs_dir, docs_pickle_file_name)with open(docs_pickle_file_path, "rb") as p_file: text_embedding_metadata_triplets = pickle.load(p_file)#2.3 Set up the embeddings modelembeddings = OpenAIEmbeddings()#3.0 Setting up the retrieverfaiss_vectorstore = FAISS.from_embeddings(text_embedding_metadata_triplets["embeddings"], embeddings,text_embedding_metadata_triplets["meta"])retriever = faiss_vectorstore.as_retriever() Building the question-answering componentThe RAG question-answering promptThe question-answering component has a system prompt and a user prompt. The system prompt and the user prompt template are shown below. They are stored in prompts.py.

```

This is the prompt for the RAG rag_system_prompt = """You are helping me find my way through the Semaphore CI documentation. You answer my questions based on the context. And then you direct to the source of the document, where I can read more.You must answer based on the context only. Do not use your general knowledge.If you don't know the answer then reply with "I don't know".You are always very optimistic and encouraging."""rag_user_prompt = """The question is in the "question" XML tags.The context is in the "context" XML tags.The context is made up of documents. Each "document" is delimited by its own XML tag.Each document has a "source" property. You need to direct me to the source of the document. Because the source of the document will have more details.Answer the question based on the context. If you don't know the answer then reply with "I don't know".{context}{question}The question is in the "question" XML tags.The context is in the "context" XML tags.The context is made up of documents. Each "document" is delimited by its own XML tag.Each document has a "source" property. You need to direct me to the source of the document. Because the source of the document will have more details.Answer the question based on the context. If you don't know the answer then reply with "I don't know".Use the following JSON template for your reply. Write valid JSON.{{ "answer": Answer the question. Make sure that you explain where I can find more information - in the source of the relevant documents. "sources": [Output the sources of the documents that you used to answer the question into this array.]}}"""

``` How to use the LLM to answer the questionThe retriever retrieves the documents that are relevant to our question. Then we need to insert those documents into the LLM’s prompt. The inserted documents are called the context. The prompt instructs the LLM to answer the question, but only based on the context in the prompt.

We need to concatenate all of the retrieved documents into one string. And the final string must be in a style that matches our prompt. We will write a function called format_docs(). You can see it in the code below. Note how we are using the metadata as well.

import promptsfrom textwrap import dedentfrom langchain_openai import ChatOpenAIfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.runnables import RunnablePassthrough#0. Function used to arrange the search results. Could be used in other code.def format_docs(docs): """ This function formats the documents for insertion into the RAG prompt """ doc_template_string = dedent(""" <document source="{the_source}"> {content} </document>""") return "\n\n".join(doc_template_string.format(content=doc.page_content,the_source=doc.metadata["source"]) for doc in docs)#1. Setupmodel_to_use = "gpt-4o-mini"#...The retriever parts of the file have been omitted for clarity#4.0 Setting up the RAG chainrag_chain = ( {"context": retriever | format_docs, "question": RunnablePassthrough()} | our_prompt_template | model)#5.0 Calling the RAG chain in a loop to answer question in the terminalwhile True: user_question = input("Your question: ") rag_answer = rag_chain.invoke(user_question) print("\nBot's answer: {0}\n".format(rag_answer["answer"])) print("For more details see: {0}\n\n".format(rag_answer["sources"])) The complete RAG chatbot fileThe retriever code and the question-answering code are part of a file called rag_bot.py. Here’s the complete file.

```

This file has the chatbotimport osimport pickleimport promptsfrom textwrap import dedentfrom langchain_openai import ChatOpenAIfrom langchain_openai import OpenAIEmbeddingsfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_community.vectorstores import FAISSfrom langchain_core.runnables import RunnablePassthrough#0. Function used to arrange the search results. Could be used in other code.def format_docs(docs): """ This function formats the documents for insertion into the RAG prompt """ doc_template_string = dedent(""" {content} """) return "\n\n".join(doc_template_string.format(content=doc.page_content,the_source=doc.metadata["source"]) for doc in docs)if name == "main": # Optional: Prompt the user for the API key #import getpass #os.environ["OPENAI_API_KEY"] = getpass.getpass() #1. Setup model_to_use = "gpt-4o-mini" docs_dir = "summarized_docs/" docs_pickle_file_name = "texts_and_embeddings.pkl" docs_pickle_file_path = os.path.join(docs_dir, docs_pickle_file_name) with open(docs_pickle_file_path, "rb") as p_file: text_embedding_metadata_triplets = pickle.load(p_file) #2.0 Setting the Langchain objects ##2.1 Setting up a Langchain "model" object model = ChatOpenAI(model=model_to_use).with_structured_output(None, method="json_mode") ##2.2 Setting up our prompt template our_prompt_template = ChatPromptTemplate.from_messages( [ ("system", prompts.rag_system_prompt), ("user", prompts.rag_user_prompt)] ) #2.3 Set up the embeddings model embeddings = OpenAIEmbeddings() #3.0 Setting up the retriever faiss_vectorstore = FAISS.from_embeddings(text_embedding_metadata_triplets["embeddings"], embeddings,text_embedding_metadata_triplets["meta"]) retriever = faiss_vectorstore.as_retriever() #4.0 Setting up the RAG chain rag_chain = ( {"context": retriever | format_docs, "question": RunnablePassthrough()} | our_prompt_template | model ) #5.0 Calling the RAG chain in a loop to answer question in the terminal while True: user_question = input("Your question: ") rag_answer = rag_chain.invoke(user_question) print("\nBot's answer: {0}\n".format(rag_answer["answer"])) print("For more details see: {0}\n\n".format(rag_answer["sources"]))

`` Putting it all togetherDirectory structureWe have three.pyfiles. Our three prompts go intoprompts.py.summarize_docs.pypreprocesses the documents.rag_bot.py` is the RAG chatbot that runs in terminal.

I have drawn the directory structure below. The semaphore folder is a clone of the new Semaphore git repo. I haven’t shown all of the files. We are working with the markdown files in the docs directory.

All of the paths that the chatbot displays are relative to semaphore/docs/. There is a second docs directory within semaphore/docs/. So semaphore/docs/docs/getting-started/guided-tour.md will be displayed as docs/getting-started/guided-tour.md.

📂Top_level_dir ┣ 📂docs_chatbot ┃ ┣ 📂summarized_docs ┃ ┃ ┗ 📜texts_and_ebeddings.pkl <== This file is generated by summarize_docs.py ┃ ┣ 📜__init__.py <== Optional. ┃ ┣ 📜prompts.py ┃ ┣ 📜rag_bot.py ┃ ┣ 📜requirements.txt ┃ ┗ 📜summarize_docs.py ┗ 📂semaphore <== This is a clone of the repo at https://github.com/semaphoreci/semaphore.git ┃ ┣ 📁.git ┃ ┣ 📁.github ┃ ┣ 📁.semaphore ┃ ┣ 📁docs <== This is the docs directory that we are working with ┃ ┣ 📜LICENSE ┃ ┗ 📜README.md Setting the OpenAI API KEYI have set the OpenAI API KEY as an evironment variable on my machine. I use the following code in the terminal. Note how I clear history.

export OPENAI_API_KEY="my_api_key_here_123"history -c Then I press ctrl+shift+L to clear the terminal window. Your keyboard shortcuts might be different.

Another option is for the python code to prompt the user for the API key. You can find some commented out code in summarize_docs.py and rag_bot.py.

```

Optional: Prompt the user for the API keyimport getpassos.environ["OPENAI_API_KEY"] = getpass.getpass()

`` Optional code that writes the summmaries and augmented context to disksummarize_docs.py` has some optional code that writes the summmaries and augmented context to disk. You can remove it if you don’t want it. These files are a copy of the text that we are storing in our retriever.

```

3.5 Optional: Write the sumarised file to disks_file_name = os.path.join(output_dir, our_path.replace("/","_"))with open(s_file_name, "w") as s_file: print(s_file_name) path_spec_text = "Path to file: {0} \n\n".format(our_path) s_file.write(the_summary)

``` Running the chatbotHere are the shell commands to preprocess the documents and run the RAG chatbot. I have put the code for this chatbot into a public git repo. So you can set up all of the files by just cloning two git repos.

git clone https://github.com/semaphoreci/semaphore.gitgit clone https://github.com/slavarazbash/docs_chatbot.git Then you will need to install the required packages. I recommend using a virtual environment.

CD docs_chatbotpython -m venv myenvsource myenv/bin/activatepip install -r requirements.txt These are the commands to preprocess the documents and start the chatbot.

python summarize_docs.pypython rag_bot.py Try it yourself. Please reach out if you have any questions or difficulties.

The post How to Build a RAG Chatbot to Chat with Documents Using Sparse Data appeared first on Semaphore.

View Details

As web users, we have all encountered “404” or “500” error pages at least once. Ever wondered what these digits mean? Well, those numbers are not casual and are known as HTTP status codes!

These are designed to communicate status information between servers and clients. While they appear as seemingly innocuous codes, attackers leverage them for vulnerability exploitation in backend applications.

In this guide, you will understand what information these HTTP status codes provide and how attackers exploit them to compromise your server.

Let’s dive in!

What Are HTTP Status Codes?HTTP response status codes are three-digit numbers that the server sets in the response to a client request. They represent a way to classify the outcome of an HTTP request and can be divided into five classes:

• 1xx: Informational responses
• 2xx: Successful responses
• 3xx: Redirection messages
• 4xx: Client error responses
• 5xx: Server error responses

Specifically, HTTP error codes are all status codes that go from 400 to 599. These refer to situations where the request could not be successfully processed because of a client or server error.

Each status code provides different information to the caller, and it is up to the backend developer to set the right HTTP status error code for each particular error. For instance, a 400 Bad Request response indicates an invalid request from the client, while a 500 Internal Server Error signals a generic application-level error in the server.

Can Attackers Exploit HTTP Status Codes?TL;DR: Yes, HTTP error codes are generally exploited by attackers to perform sophisticated attacks on servers.

HTTP status codes provide information about the success or failure of a client’s request to a server. This is free information for attackers, who can monitor and study changes in HTTP response status codes to forge increasingly effective requests. This iterative process enables them to slowly bypass web application security measures and get closer to their malicious objectives.

For instance, attackers could build a bot that performs a brute force attack to automatically find all resources exposed by your server. Initial requests may result in 404 Not Found errors, but subsequent requests might produce 401 Unauthorized, 403 Forbidden, or even 200 OK responses, indicating public endpoints to attack.

Status codes, whether success or error, help attackers map out a backend application’s routes and identify potential security flaws. They reveal the server’s behavior in response to specific inputs, which is awesome feedback for attackers!

Why Returning Wrong HTTP Status Codes on Purpose Is Not A SolutionPrograming a backend application to return “wrong” HTTP status codes on purpose is a possible STO (Security Through Obscurity) technique. The idea is to make the public behavior of a server application more opaque as a way to enhance its security.

Setting unexpected HTTP status codes will confuse attackers and could disrupt most automated attacks, slowing down the progress of malicious users. At the same time, this tactic will confuse developers as well. Deviating from standardized HTTP error code conventions will also complicate troubleshooting and debugging. Plus, this STO strategy cannot be applied on public endpoints, where adherence to standards is crucial.

While returning uncommon HTTP response status codes may momentarily disorient attackers, it does not address the underlying vulnerabilities in the server. In other words, it is just a temporary deterrent that does not really mitigate attack vectors. Here is why you should not rely on such an exotic approach to web application security.

Most Interesting HTTP Status Code for AttackersTime to dig into the most interesting HTTP status codes that attackers track for malicious purposes. Let’s explore the common attacks associated with these response codes!

200The HTTP 200 OK success status response code indicates that the server has successfully processed the request. While this status code does not carry much information by itself, it signals to attackers that their attempt—whether automated or manual—was successful. In essence, it tells attackers that they have found a way to legitimately receive responses from the server. That opens the door to a world of possibilities, from data scraping to unauthorized access.

HTTP 200 OK responses are especially valuable to attackers when they target endpoints that should be protected by authorization. The Common Weakness Enumeration (CWE) describes this scenario as “Missing Authorization.”

For example, a 200 OK response from an unknown IP on paths like /admin/ or /private/ is highly interesting to a user with malicious purposes, as it suggests that these sensitive areas of the server are accessible without proper authorization.

400The 400 Bad Request status code indicates that the server cannot process the request due to a client error. This is generally caused by an invalid request message framing, deceptive request routing, or malformed request.

Attackers can use the 400 status code to test the robustness of a web application. By deliberately sending malformed or unexpected data in a process known as fuzzing, they can observe how the server handles such inputs. If the web application returns a 400 status code, it confirms that some input validation is in place. However, it might expose weak or inconsistent input validation behavior, which could be exploited further.

Additionally, 400 responses could include detailed error messages about the invalid input. These messages can inadvertently reveal sensitive information such as variable names used in the business logic, the data validation library in use, or even database schema details. Attackers can employ all that information to craft more specific attack vectors. This is why 400 error messages should always be as generic as possible.

A 400 status code can also be harnessed to understand whether an endpoint exists, but the request was improperly formed. This information is useful to discover API endpoints and resources on a server.

401The 401 Unauthorized status code indicates that the request has not been processed because it lacks valid authentication credentials for the target resource. Attackers typically send automated requests to various endpoints, looking for 401 responses. This helps them identify that an endpoint exists and requires authentication.

Once an attacker identifies the existence of an authenticated endpoint, they can initiate a brute force attack by systematically trying different username and password combinations. In some cases, backend applications return different 401 responses based on whether the username or password was incorrect (e.g., “Invalid username” vs “Invalid password”). Malicious users can exploit this to detect valid usernames. Instead, send fewer details back with the 401 (e.g. “Unknown username and/or incorrect password”).

401 responses are also valuable for studying the type of authentication (e.g., basic auth, bearer tokens, etc.) used by the server and the required parameters to satisfy it. With this information, it is possible to craft more sophisticated attack vectors aimed at exploiting weaknesses in the security authentication system of a web application.

403The 403 Forbidden error code indicates that the server understood the request but refused to authorize it. Attackers leverage this status code to identify which resources are restricted by the server. The next step involves attempting to gain unauthorized access to these resources by modifying cookies, forging authentication tokens, or tampering with URL parameters.

By analyzing how the server responds to different variations of the request, attackers may also gain insights into the application’s logic, potentially spotting weaknesses in the access control mechanism.

A 403 response could also indicate that directory listing is disabled, but the directory itself exists. This may inadvertently disclose the existence of files or directories that attackers can then target for further vulnerability exploitation.

404The 404 Not Found status code is likely one of the most common HTTP error codes for attackers, indicating that the server cannot find the requested resource.

By utilizing directory traversal or brute-force attacks based on 404 responses, attackers can trace out the server’s directory structure. A similar mechanism can be employed for performing endpoint and/or file enumeration. Thus, 404 error codes are useful for identifying which endpoints and resources exist and are accessible on a server.

In most cases, 404 responses consist of a generic HTML page that provides more information than just the basic “Not Found” message. These predefined web pages may also provide enough information for attackers to make informed guesses about the technology stack in use by the server. The solution here is to always override these default pages.

405The 405 Method Not Allowed status code indicates that the server does not support the HTTP method of the request.

Discovering the existence of an API endpoint means knowing its path and HTTP method (e.g., GET, POST, PUT, PATCH, or DELETE). 405 responses help an attacker find the right HTTP method after identifying a valid path the server listens to.

To hinder the attacker, in the 405 response message, do not specify the correct HTTP method the caller should use to reach the API endpoint.

429The 429 Too Many Requests error code indicates that the client has sent too many requests to the server in a given amount of time. That means that rate-limiting measures are in place. Attackers will then adjust their attack strategies accordingly, such as routing automated requests through a residential proxy network to get fresh exit IPs and avoid blocks.

Additionally, 429 responses may include RateLimit-* or X-RateLimit-* headers. These headers inform the client about the number of requests allowed in a given time frame, which is key information for attackers. Also, based on the names of those headers, it may even be possible to guess the library employed to implement rate limiting and consequently the technology stack in use.

The solution here is to respond with a 429 status code, without including rate-limiting headers or specific information about how your server limits incoming requests. In the case of requests routed by proxy servers, there is not much you can do, especially if they provide genuine IPs from residential devices. However, if the proxy IPs come from datacenters, their numbers are usually incremental, so they can be detected and banned in bulk.

500The 500 Internal Server Error status code is the most generic among HTTP error codes and indicates that the server failed to fulfill the request. The reason behind a 500 response may be countless, but responses might include error messages or even entire stack traces.

Those error messages typically provide useful information about the configuration or internal workings of the server. The ultimate goal of attackers is to use that information to identify vulnerabilities to exploit.

Also, keep in mind that backend frameworks and libraries typically return generic 500 Internal Server Error responses in case of server errors. The default messages in these responses can sometimes be enough to guess the server’s technology stack.

To avoid assisting the attacker, override the default message and always respond with generic errors. For example, a message like “Internal Server Error” is enough to explain any 500 errors.

Resources to Strengthen Your Security GameSecurity is a cat-and-mouse game, and if you want to stay ahead, continuous learning is essential. Equip yourself with these valuable resources:

• OWASP Secure Coding Practices: A comprehensive list of community-backed best practices designed to protect against the most common security attacks. This guide is supported by The OWASP Foundation, a nonprofit organization focused on improving software security.
• TryHackMe: A collection of hands-on labs that teach you how to attack web applications through interactive, real-world exercises. By stepping into an attacker’s shoes, you can gain a deeper understanding of their tactics and strategies.
• SANS Cyber Security Training: It offers a wide range of courses and certifications covering various aspects of cybersecurity, from foundational concepts to advanced threat detection techniques. Their training is widely recognized and respected in the industry.

ConclusionIn this guide, you learned what HTTP codes are and how attackers can use them to carry out targeted attacks against your server or web application. An idea to avoid this problem is to start returning uncommon HTTP status codes on purpose. While that may seem like an effective approach against vulnerability exploitation, it is not a real solution. The real solution is to follow API security best practices.

The post How Attackers Use HTTP Status Codes for Malicious Purposes appeared first on Semaphore.

View Details

The cost of discovering flaws and security vulnerabilities in a production application can be staggering. When issues arise post-deployment, they can significantly damage your business’s reputation and undermine user trust. Given these challenges, many teams are adopting shift-left testing tools to enhance quality assurance early in the development process.

By moving testing activities “left” in the development timeline, you can identify and remediate issues before they escalate. This leads to high-quality, more stable, and secure applications.

In this article, you will understand how the shift-left testing approach works and then examine some of the best tools to implement it.

Let’s dive in!

What Is Shift-Left Testing?Shift-left testing is a practice that emphasizes moving testing activities early in the development process. By “shifting left” testing, developers and QA engineers can identify and fix issues and bugs sooner, when they are easier and cheaper to address:

This testing methodology typically happens during the design and coding phases of the software development lifecycle. It involves designing and implementing software that works as expected by following methodologies like Test-Driven Development (TDD).

Along with unit tests, integration tests, and E2E tests, shift-left testing also uses static code analysis (SCA) solutions. These tools scan code, configurations, dependencies, and settings in your repository to detect security vulnerabilities and other flaws. The ultimate goal is to ensure a more robust and reliable application before it reaches production, thereby improving the overall quality assurance (QA) process.

This approach to testing contrasts with shift-right testing, which focuses on testing later in the software lifecycle, closer to production. While shift-right testing relies on techniques like application monitoring to find bugs in live systems, shift-left testing occurs before the application is fully deployed and running.

Main Benefits of Shift-Left TestingShifting testing to the left in your software development cycle leads to several benefits, including:

• Quick bug detection: Identify flaws as early as possible in the development cycle, allowing you to address and easily fix them before they become too big and complex. By preventing the accumulation of unresolved issues that can lead to long-term problems in your application, the shift-left testing approach reduces technical debt.
• Improved software quality: Addressing vulnerabilities, code smells, and bugs before they reach production results in higher-quality code. A live application with fewer issues is also a more reliable product for your users.
• Lower costs: Addressing issues earlier in the development process is generally much cheaper than identifying and resolving them later in production. Bugs can significantly slow down and frustrate developers, so facing them timely speeds up coding. The result is a quicker time-to-market, which ultimately reduces overall development costs.
• Enhanced collaboration: Placing so much importance on testing from the beginning encourages developers and QA teams to collaborate closely from day one. This fosters a synergetic environment among development, security, and quality teams.
• Strengthened security: Continuous analysis of code, configurations, and dependencies helps mitigate security risks before it is too late. For more guidance, read our detailed guide on security best practices.

Recommended Shift-Left Testing ApproachShift-left testing is a process that encompasses several facets of software testing, such as:

• Manual tests: Scripts written by developers to validate the logic and functionality of the code. These include unit tests, performance tests, integration tests, automated tests, and more.
• Dependency inspection: Examine third-party libraries and packages your application relies on to identify known vulnerabilities and make sure they are up to date.
• Configuration inspection: Verify adherence to security best practices by evaluating the settings of the libraries and technologies used by your application.
• Code quality analysis: Analyze source code for quality issues, such as using variables instead of constants, and potential security vulnerabilities, such as SQL injection, without executing it.

The recommended shift-left testing approach is to adopt a set of dedicated tools to cover each of the aspects mentioned above. This ensures thorough coverage and enables you to efficiently and effectively achieve all the benefits introduced earlier.

Keep in mind that—apart from manual tests written by your developers—this approach to testing is mainly carried out by static code analysis (SCA) tools. In particular, shift-left testing tools automatically inspect your codebase, examining all its files for bugs, flaws, and security vulnerabilities.

Since your code or configurations may contain credentials or sensitive information related to your business, those automated tools should be open-source. This way, the community can verify how they operate and make sure they are not stealing data or engaging in other misconduct.

Top 5 Open-Source Shift-Left Testing ToolsExplore some of the best and most useful shift-left testing tools available on the market. For each tool, we will analyze these aspects:

• Testing scenarios covered: Manual testing, dependency inspection, static code analysis, etc.
• Supported programming languages: The programming languages that the tool supports.
• Types of applications supported: Web applications, desktop applications, mobile apps, etc.
• GitHub stars: The number of stars the project has received on GitHub.
• Downloads: The number of times the tool has been downloaded from dependency registries like npm.
• Pros: The main advantages of the tool.
• Cons: The main disadvantages of the tool.
• Pricing: The cost of the tool for business or enterprise needs, if applicable.

If you are eager to see how these tools compare, take a look at the summary table below:

| Tool | Goal | Supported Programming Languages | Supported Applications | GitHub Stars | Downloads | Open-Source Version | Premium Version | | --- | --- | --- | --- | --- | --- | --- | --- | | Automated Test Frameworks | Write and run test scripts | Depends on the framework (e.g., JavaScript, Python, Java, etc.) | Web, desktop, mobile applications | Varies | Varies | ✅ | Varies | | OSV-Scanner | Vulnerability scanning | C, C++, Dart, Elixir, Go, Java, JavaScript, PHP, Python, R, Ruby, Rust | Web applications | 6k | Undisclosed | ✅ | ❌ | | TruffleHog | Secret scanning and detection | Any | Any | 15.8k | Undisclosed | ✅ | ✅ | | Snyk | Vulnerability scanning across code, dependencies, containers, and IaC | Bazel, C/C++, Dart, Elixir, Go, Java, Kotlin, JavaScript, .NET, PHP, Python, Ruby, Swift, TypeScript, VB.NET | Web, desktop, mobile applications | 4.9k | 550k+ weekly | ✅ | ✅ | | Checkov | Misconfiguration scanning for IaC | Any | Web applications | 7k | 1+ million weekly | ✅ | ❌ |

Time to dig into these tools (in random order)!

Automated Test Frameworks

More than a specific shift-left testing tool, this represents a category of tools. Specifically, it encompasses all frameworks you can use to write manual test scripts in your codebase. Examples of such solutions include Selenium, pytest, Playwright, Jest, Mocha, and JUnit.

In most cases, these technologies come with everything you need to implement one or more specific types of tests in your application. They offer an API for writing tests, assertion libraries, test runners, test reporting capabilities, and additional features.

For additional details, check out our software engineering testing guides.

🧪 Testing scenarios covered: Manual testing via unit tests, integration tests, smoke tests, component tests, E2E tests, and more

💻 Supported programming languages: Depends on the chosen tool, but there is at least one testing framework available for all major programming languages.

🌐 Types of applications supported: Web applications, desktop applications, mobile apps, and more

⭐ GitHub stars: Some of these tools, like Playwright, have over 65k stars

📥 Downloads: Some tools, such as Jest, have over 24 million weekly downloads

👍 Pros:

• All-in-one testing solutions that provide complete testing capabilities
• Integration with CI/CD pipelines
• Strong community support

👎 Cons:

• Some tools have a steep learning curve

💰 Pricing: The great majority of these tools are free, but some offer advanced features for a fee. For example, Cypress comes with a solution with premium features called Cypress Cloud

OSV-ScannerOSV-Scanner is a vulnerability scanner developed by Google in Go that uses data provided by OSV.dev, a distributed open-source vulnerability database.

OSV-Scanner analyzes your project’s list of dependencies, connects to the OSV.dev database, and reports any vulnerabilities that affect them. Compared to scanners based on closed-source databases, the open nature of OSV.dev ensures that each advisory is sourced from a reliable and public platform. Also, this guarantees that anyone can contribute to these advisories.

🧪 Testing scenarios covered: Dependency inspection for detecting security vulnerabilities

💻 Supported programming languages: C, C++, Dart, Elixir, Go, Java, JavaScript, PHP, Python, R, Ruby, and Rust

🌐 Types of applications supported: Web applications

⭐ GitHub stars: 6k stars

📥 Downloads: Undisclosed

👍 Pros:

• Work with multiple programming languages and package indexes
• Developed and maintained by Google
• Based on an open-source and distributed database of vulnerabilities

👎 Cons:

• Over 100 issues currently reported on GitHub

💰 Pricing: Free

TruffleHogTruffleHog is a secret scanning tool that searches deep into code repositories, version histories, S3 directories, Docker images, and more to detect hidden secrets like passwords, API keys, and sensitive credentials.

TruffleHog also features preventive measures, using hooks to stop secrets from being committed, and offers remediation tools with alerts and guides. It minimizes false positives by validating detected credentials.

🧪 Testing scenarios covered: Secret detection

💻 Supported programming languages: Any

🌐 Types of applications supported: Any

⭐ GitHub stars: 15.8k stars

📥 Downloads: Undisclosed

👍 Pros:

• GitHub, S3, directory, GCS, and Docker scanning with 800+ secret detectors available
• Credential verification to avoid false positive
• Multi-branch analysis capabilities

👎 Cons:

• Most powerful features are available only in the premium Enterprise version

💰 Pricing:

• Open-Source: Free
• Enterprise: Premium features available at an undisclosed price (contact the support team for pricing)

SnykSnyk is a developer-first, cloud-native security platform that helps identify and fix vulnerabilities across various aspects of software development. The products it offers are :

• Snyk Open Source: Detect and fix open-source vulnerabilities.
• Snyk Code: Identify and resolve security issues in application code.
• Snyk Container: Secure container images and Kubernetes applications.
• Snyk Infrastructure as Code: Address insecure configurations in Terraform and Kubernetes code.

You can integrate the Snyk CLI into your workflow, IDE, or CI/CD pipeline for seamless security testing.

🧪 Testing scenarios covered: Vulnerability detection in dependencies, code, containers, and configurations

💻 Supported programming languages: Bazel, C.C++, Dart and Flutter, Elixir, Go, Java, Kotlin, Javascript, .NET, PHP, Python, Ruby, Swift, Typescript, and VB.NET

🌐 Types of applications supported: Web applications, desktop applications, and mobile apps

⭐ GitHub stars: 4.9k stars

📥 Downloads: 550k+ weekly downloads

👍 Pros:

• Support for multiple shift-left security tools
• Simple CI/CD integration
• Dedicated plugins for integration directly in popular IDEs like Visual Studio Code, Intellij IDEs, and Eclipse

👎 Cons:

• Requires a Snyk account to work
• Not a fully open-source vulnerability database

💰 Pricing:

• Free: For individual developers and small teams (free forever)
• Team: For development teams looking to build security (contact sales for pricing)
• Enterprise: For standardizing developer-first security across an organization (contact sales for pricing)

CheckovCheckov is an open-source security tool that scans cloud infrastructure configurations for misconfigurations. It helps identify security and compliance issues in Infrastructure as Code (IaC) platforms, preventing potential vulnerabilities from being introduced into cloud environments.

The IaC providers supported by Checkov features Terraform, Terraform plan, CloudFormation, Kubernetes, ARM Templates, Serverless, Helm, and AWS CDK.

In addition to misconfiguration scanning, Checkov can also detect vulnerabilities in open-source packages and container images.

🧪 Testing scenarios covered: Code misconfiguration detection, dependency vulneraribility detection

💻 Supported programming languages: Any

🌐 Types of applications supported: Web applications

⭐ GitHub stars: 7k stars

📥 Downloads: 1+ million weekly downloads

👍 Pros:

• Comes with over 1,000 built-in policies
• Easy integration into existing developer workflows
• Trusted by Prisma Cloud Application Security

👎 Cons:

• Requires Python to work

💰 Pricing: Free

Honorable MentionsOther shift-left testing tools worth mentioning are:

• Trivy: A security scanner that checks for vulnerabilities in container images, file systems, and Git repositories. It can also execute IaC scanning to detect misconfigurations. Discover more in our tutorial on continuous container vulnerability testing with Trivy.
• SonarQube: A tool for continuous code quality and security inspection. It analyzes source code to detect code smells, bugs, and security vulnerabilities across multiple programming languages. Find out more in our guide on SonarQube.

ConclusionIn this guide, you learned about the importance of integrating testing as early as possible in the development lifecycle. As demonstrated, this methodology offers significant benefits, such as reduced costs and faster release times. However, achieving these advantages requires the right tools. Here, you discovered the best open-source shift-left testing tools to facilitate the automation of testing in applications still in development.

The post Best Shift-Left Testing Tools To Improve Your QA appeared first on Semaphore.

View Details

Open-source software is the building materials of the vast majority of digital technology as we know it. Anyone can access and contribute to it, which is great for flexibility and cost, but also means there’s a risk of hidden flaws. In this episode, Stacklok’s CEO Craig McLuckie shares his tools to check the quality and security of open-source software production. Working together as a community, we can make open-source software safer and more reliable for everyone.

Edited transcription

As CEO and co-founder of Stacklok, a software company specializing in supply chain security for open-source software, Craig McLuckie has closely witnessed a shift in the nature of hackers’ methodologies. Accidental vulnerabilities, in which “someone just didn’t anticipate a buffer overflow here, bad stuff happens there,” have given up their place to “a world where hostile actors are actually trying to manipulate the supply chain.” Consequently, security is moving from finding and fixing vulnerabilities from external threats to preventing malicious contributors from creating, nurturing, and preserving vulnerabilities.

With the recent antecedent of the XZ Utils backdoor as the incident of the highest profile, Craig points out that despite the “high importance, high repackaging, high reuse” of open-source “critical supporting technologies,” these, in turn, have “relatively low maintainer engagement.” As the sophistication, dedication, and scale of this attack raise strong suspicions of state-backed actors behind them, these vulnerabilities are not easy to detect, nor to determine their reach: “With these things, you never know what the blast radius is really,” Craig adds.

The ubiquity of open-source softwareSecurity threats of the magnitude of the XZ vulnerability put under the spotlight one of the main positive qualities of open-source software: Collaboration. Fair enough, it may lead to fear and distrust in the open-source community, as free comes with a cost.

However, according to Craig, there’s no such thing as “non-open source” in today’s industry, in which virtually all software relies on open-source components at some level. Whether it’s the programming languages themselves (like Java or Python) or the underlying libraries, frameworks, and operative systems, “open source is basically the underpinning of everything that we build.”

Moreover, Craig argues that when companies choose to work exclusively with proprietary vendors, what they’re really doing is engaging in a “risk adjustment process” or “arbitraging risk.” In other words, these companies aren’t actually avoiding open source; they’re merely shifting the responsibility for managing open source risks to their vendors. However, according to him, this perspective is flawed in as much as the lack of transparency in these vendor relationships doesn’t reduce risk exposure.

The SolarWinds incident, in which a provider of system management tools for network monitoring was compromised by attackers’ malicious code inserted into updates of its platform, is an example of how even when companies choose to work exclusively with proprietary vendors, they are still exposed to the risks associated with open-source software. Even when working with a closed-system vendor, that vendor is still consuming open-source technology—but the customer doesn’t necessarily know what components are being used, how they’re being used, or where they originate from. Hence, these companies are essentially choosing to close their eyes and trust that vendors are behaving responsibly, rather than actively managing these risks themselves.

Stacklok: Addressing supply chain security“Collaboration just goes away when there’s fear in the ecosystem,” Craig affirms. Still, he is confident that solving the complex issues of software supply chain security requires collaboration from many parties, not just a single vendor or company. Aligned with this philosophy, Craig envisions Stacklock as a “seed crystal” fostering an active community rather than remaining a solitary solution provider.

Stacklok was born out of Craig’s firsthand experience with enforcing supply chain security standards and controls at large tech companies like Google and VMware. Recognizing this as a widespread issue in the open-source community, Craig sought a solution that would provide greater transparency and traceability. Exploring the potential of employing a generic system for labeling software resources, he eventually came across Luke Hein, the creator of Sigstore. Sigstore’s innovative approach to software signing, which “connects a piece of software with information about its inception,” aligned perfectly with Craig’s vision of empowering users to understand the origins and trustworthiness of the software they rely on.

Setting up policies with MinderThe idea behind Stacklok was to give users ”the ability to apply policy” at any stage of the software development life cycle, ensuring that only software meeting predefined standards would integrate into their projects. To this end, Stacklok’s platform Minder enables multiple controls across the entire software development and deployment process. For example, setting policies around Git repositories, to each merge request, container registries, and Kubernetes deployments.

If you’re building software with dependencies, Craig recommends enforcing a policy “that blocks a merge request if there’s a vulnerability” or unfavorable attribute in it, like “bad licensing material, bad reputational scores, or with known CVEs.” Securing dependencies can save developers countless headaches. ”The last thing you want to do is discover a vulnerability and call a scan in production,” he says. As such, he calls to “intercept these things as quickly and early as possible, ideally in the IDEs.”

Minder can also wrap policies around Github actions and repositories, “and set up controls so that your organization can use or not use actions based on where they’re sourced from.” What’s more, you can write your own rules using YAML or Rego policy language.

Judging dependencies with TrustyWhile Minder is useful for setting up granular policies, Stacklock’s second product, Trusty, employs sophisticated data science techniques to assist organizations in evaluating the sustainability of software. By providing “high-quality signals intelligence,” Craig explains, Trusty empowers organizations to assess their hypotheses and make informed decisions about the software they adopt, proactively identifying potential sustainability issues before they escalate.

Signals are pieces of information that provide insight into the quality, security, or reliability of a software package or component. According to Craig, it is necessary to not take signals as a given, but to enforce a signaling process or source we can trust: “Unless you actually have a signature, unless you have some firm point of attestation that an identity that’s known to be associated with this package is the identity that published the package and published the signature of the package, you just don’t know.”

Craig asserts that a technology like SIGSTOR is essential for determining the responsible sourcing of the products or services we consume. However, he acknowledges that implementing such technology can be more challenging in open-source projects. In this way, even in the absence of SIGSTOR, Craig believes that Trusty valuable intelligence signals can assist in assessing responsible sourcing.

For signaling packages, Trusty uses historical provenance. Trusty examines the repo and checks associated release tags to find a chain of trust back to the source code: “If the tags and the sort of known versions associated with this package lineup is probably the package you’re looking for,” Craig explains. “If they don’t, it probably isn’t—we would then give it a very low score because we don’t have proof of its historic provenance.”

Trusty also performs principal component analysis, which analyzes a database of bad packages and compares their composition with good packages. According to Craig, this analysis allows users to “build a statistical model so that when we’re looking at something that we’ve never seen before, we can sort of attest whether it’s good or bad.” Trained AI can find patterns that can be correlated with packages falling out of reputation, such as a lack of maintainers. In these cases, the platform will recommend better alternatives.

The bottom lineStacklok’s roadmap includes new features for private repositories and introduces commercial offerings. Also, while currently focused on GitHub integration, Stacklok is working on expanding to other providers, such as OCI registry provider and Kubernetes provider to continue introducing control points across various stages of the software development lifecycle.

Visit Stacklok.com and github.com/stacklok to learn more about the project and test its product. Join Stacklok’s Discord community to chat directly with its engineers and find out the latest about its products’ development.

Follow Craig on X at @cmcluck. To learn more about his views on supply chain security, check his talk at the Open Source Software Summit Seattle 2024.

The post Craig McLuckie on Enforcing Compliance on Open Source Development appeared first on Semaphore.

View Details

Keeping users engaged is crucial to any application’s success. Automated email summaries provide an effective way to achieve this by delivering curated content that keeps users connected to your platform. In this tutorial, we’ll guide you through building an AI-enabled automated email summary system with continuous integration and deployment (CI/CD) using Semaphore.

PrerequisitesBefore starting this tutorial, you should have:

• Node.js version 20 or higher
• GitHub account for version control
• Semaphore account for CI/CD setup
• Anthropic account for API access to Claude
• Ethereal Email account for creating a test email account for mocking

All of the code for this article is available at ajcwebdev/semaphore-ai-email. Begin by cloning this repo, creating a .env file, and installing the project’s dependencies:

git clone https://github.com/ajcwebdev/semaphore-ai-emailcd semaphore-ai-emailcp .env.example .envnpm i Step 1: Setting Up the Email Service and TemplatesWe’ll begin by creating the foundation of our email summary system: the email service and templates. This step involves setting up Nodemailer for email sending and designing responsive HTML templates for our summaries.

First, open the file named send-email.js. This script will handle the email sending process:

// send-email.jsimport nodemailer from 'nodemailer'import { fileURLToPath } from 'url'import { dirname, join } from 'path'import fs from 'fs/promises'const __filename = fileURLToPath(import.meta.url)const __dirname = dirname(__filename)async function sendEmail() { let transporter = nodemailer.createTransport({ host: process.env.EMAIL_HOST, port: 587, auth: { user: process.env.EMAIL_USER, pass: process.env.EMAIL_PASS } }) const htmlTemplate = await fs.readFile(join(__dirname, 'email-template.html'), 'utf-8') let mailOptions = { from: `"Test Email" <${process.env.EMAIL_USER}>`, to: process.env.EMAIL_USER, subject: 'Weekly Update', html: htmlTemplate } let info = await transporter.sendMail(mailOptions) console.log('Message sent: %s', info.messageId) console.log('Preview URL: %s', nodemailer.getTestMessageUrl(info))}async function main() { try { await sendEmail() console.log('Email sent successfully. Waiting 1 week before sending the next one...') } catch (error) { console.error('Failed to send email:', error) } await new Promise(resolve => setTimeout(resolve, 7 * 24 * 60 * 60 * 1000))}main().catch(console.error) This script sets up a Nodemailer transporter using Ethereal Email, a catch-all email testing service. Ethereal is ideal for development as it captures emails without actually sending them, providing a preview URL for each message.

Let’s break down the key components:

• Imports and setup: We import necessary modules and set up file path utilities.
• sendEmail function: Creates a Nodemailer transporter, reads our HTML template and CSS file, and sends the email.
• File reading: We use fs.promises to read the template and CSS files asynchronously.
• Email configuration: The email is configured with a from address, to address, subject, HTML content, and CSS attachment.
• Logging: After sending, we log the message ID and preview URL provided by Ethereal.
• main function: Runs in a loop, sending an email once every week. This simulates a scheduled task, which we’ll implement properly with Semaphore later.

This script references two articles selected and saved in files called ARTICLE_1.js and ARTICLE_2.js. You can use the examples provided in my repo or replace them with your own. The files are structured like so (see example repo for entire file contents):

`` // ARTICLE_1.jsexport const ARTICLE_1 = { title: "Why you should write your own LLM benchmarks - with Nicholas Carlini, Google DeepMind", content:Today's guest, Nicholas Carlini, a research scientist at DeepMind, argues that we should be focusing more on what AI can do for us individually, rather than trying to have an answer for everyone....,}// ARTICLE\_2.jsexport const ARTICLE\_2 = { title: "Why you should write your own LLM benchmarks - with Nicholas Carlini, Google DeepMind", content:Betteridge's law says no: with seemingly infinite flavors of RAG, and >2million token context + prompt caching from Anthropic/Deepmind/Deepseek, it's reasonable to believe that \"in context learning is all you need\".

...,}** ``` **Next, open the HTML email template namedemail-template.html. The AI summaries we’ll generate in the next section are contained within themain` tags.

`` **Weekly AI Newsletter Weekly AI Newsletter Your weekly digest of AI news and insights

Top Articles Why you should write your own LLM benchmarks — with Nicholas Carlini, Google DeepMind The article features an interview with Nicholas Carlini, a research scientist at DeepMind, discussing his views on AI and his work in AI security. Key points include:

• Carlini wrote a viral blog post "How I Use AI" detailing practical ways he uses AI tools in his daily work as a security researcher. He emphasizes focusing on what AI can do for individuals rather than trying to have a universal answer.
• He created a domain-specific language for writing personalized LLM benchmarks, arguing that more people should make benchmarks tailored to their specific needs rather than relying solely on standardized tests.
• Carlini discusses his research in AI security, including work on data poisoning, model stealing, and extracting training data from language models. He emphasizes the importance of studying real-world vulnerabilities in AI systems.
• He explains his preference for focusing on attacking rather than defending AI systems, stating that he finds it more motivating and believes he can contribute more effectively by identifying vulnerabilities.

Is finetuning GPT4o worth it? — with Alistair Pullen, Cosine (Genie) This article discusses an interview with Alistair Pullen, CEO and co-founder of Cosine, about their new AI coding agent called Genie. Key points include:

• Genie achieved state-of-the-art results on the SWE-Bench coding benchmark, outperforming other solutions by a significant margin.
• Cosine worked closely with OpenAI to fine-tune GPT-4 on billions of tokens of synthetic code data, focusing on reproducing the process of how human software engineers work rather than just training on finished code.
• Genie uses a four-stage workflow: finding files, planning actions, writing code, and running tests. It can interact with GitHub and CI systems to execute and test code changes.
• The team faced challenges in areas like code retrieval, context window limitations, and balancing different programming languages in the training data.
• Cosine is looking to expand Genie's capabilities, increase their dataset size, and potentially customize models for specific customers' codebases.

© 2024 Weekly AI Newsletter. All rights reserved.

Unsubscribe | View in browser | Privacy Policy` ``` This HTML template provides the structure for our email. It includes:**

• *Responsive meta tag* for proper rendering on mobile devices
• *Header* with the newsletter title
• *Main content area* with placeholders for article summaries
• *Footer* with copyright information and standard email links

The CSS defines the styles for our email template. Key features include:

• *Media query* for responsiveness on smaller screens
• *Consistent typography* and color scheme
• *Container* with a maximum width for better readability
• *Distinct styling* for header, content sections, and footer
• *Subtle use* of background colors and padding to separate different areas of the email

In the next step, we’ll focus on content aggregation and processing, where we’ll use AI to generate summaries for our newsletter.

Step 2: Content Aggregation and ProcessingIn this step, we’ll use Anthropic’s Claude API to automatically summarize articles for our email digest. This process involves interacting with an advanced language model to generate concise, informative summaries of longer articles.

Open the file named claude.js:

**`// claude.jsimport Anthropic from '@anthropic-ai/sdk'import { ARTICLE_1 } from './ARTICLE_1.js'import { ARTICLE_2 } from './ARTICLE_2.js'const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY})async function summarizeArticle(article) { const prompt = `Please provide a one to two paragraph summary of the following article:${article}Summary:` const response = await anthropic.messages.create({ model: "claude-3-5-sonnet-20240620", max_tokens: 300, messages: [ { role: "user", content: prompt } ] }) return response.content[0].text}async function processArticles(articles) { const summaries = [] for (const article of articles) { const { title, content } = article const summary = await summarizeArticle(content) summaries.push({ title, summary }) console.log(`Summarized: ${title}`) } return summaries}async function main() { const articles = [ { title: ARTICLE_1.title, content: ARTICLE_1.content }, { title: ARTICLE_2.title, content: ARTICLE_2.content } ] try { const results = await processArticles(articles) for (const result of results) { console.log(`\nTitle: ${result.title}`) console.log(`Summary: ${result.summary}`) console.log("-".repeat(50)) } } catch (error) { console.error("An error occurred:", error) }}main()`** Let’s break down this script and examine its key components:

1. *Importing Dependencies and Articles:* We import the Anthropic SDK and two article contents from separate files. In a production environment, you might fetch these articles from a database or API.
2. *Initializing the Anthropic Client:* We create an instance of the Anthropic client, using an API key stored in an environment variable for security.
3. *Article Summarization Function:* The summarizeArticle function takes an article’s content and sends it to Claude with a specific prompt. We use the “claude-3-5-sonnet-20240620” model and limit the response to 300 tokens for concise summaries. The function returns the generated summary.
4. *Article Processing Function:* The processArticles function iterates through an array of articles, calling summarizeArticle for each one. It collects the title and generated summary for each article and returns an array of these summaries.
5. *Main Execution Function:* The main function defines an array of articles (in a real-world scenario, this data might come from an external source). It then calls processArticles and logs the results. Error handling is implemented to catch and log any issues that occur during processing.
6. *Execution:* Finally, we call the main function to start the summarization process.

Open the .env file and include your Anthropic API key:

**`ANTHROPIC_API_KEY=""`** Run the script with the following command:

**`npm run claude`** This script demonstrates using AI for content processing. By using Claude’s natural language understanding capabilities, we can automatically generate concise summaries of lengthy articles. The next step will involve setting up automated deployment with Semaphore.

Step 3: Automating the Build and Deploy Process with SemaphoreIn this step, we’ll set up a continuous integration and deployment (CI/CD) pipeline using Semaphore. This will automate our build process and schedule our weekly email digest.

The semaphore.yml file in the .semaphore directory contains the specific steps for sending the email. This allows separating the weekly email task from the regular CI/CD pipeline.

**`# .semaphore/semaphore.ymlversion: v1.0name: Send Weekly Emailagent: machine: type: e1-standard-2 os_image: ubuntu2004blocks: - name: Send Email task: secrets: - name: email-credentials jobs: - name: Send Email commands: - checkout - sem-version node 20 - npm install nodemailer - node send-email.js`** 1. *Set up Semaphore project:* To set up your Semaphore project, first create a Semaphore account, open your organization settings, and navigate to the Secrets section:

• 2. Create secrets: Click “New Secret” and name the secret email-credentials. Set your test email to EMAIL_USER, your password to EMAIL_PASS, and your email host provider to EMAIL_HOST.

• 3. Save and connect: Click “Save Secret.” Next, create a new project and connect the project to your GitHub repository:

• 4. Run workflow: This will start running your workflow:

• 5. View logs: Click “Send Email” to view the workflow logs:

• 6. Check results: Your logs should include a message saying, “Message sent” along with a preview URL. Open the preview URL to view the example email:

This separate pipeline file focuses solely on the tasks required to send your weekly email. By setting up this Semaphore configuration, you’ve automated your build process and scheduled your weekly email digest. Remember to regularly review your Semaphore logs to ensure everything is running smoothly and to catch any potential issues early.

Step 4: Analyzing and Optimizing Summary EmailsTo improve our email summaries over time, we should implement analytics to track open rates and engagement. We can set up A/B testing for subject lines and content layouts. Here’s how:

1. *Create* multiple versions of your email template.
2. *Modify* your Semaphore pipeline to randomly select a template version for each send.
3. *Use* an analytics service to track which versions perform better. Popular options include Optimizely and VWO.
4. *Periodically review* the data and update your templates based on the results.

Step 5: Scaling, Cost Optimization, and DeliverabilityAs your user base grows, consider these strategies for scaling:

• *Use* a message queue (like RabbitMQ or Redis) to handle increased load.
• *Implement* caching to reduce API calls and improve performance.
• *Optimize* cloud resource usage.

A caching mechanism can significantly reduce the number of API calls to Claude, lowering costs and improving response times.

To ensure email deliverability and prevent your domain from being blacklisted for spam, implement the following best practices:

1. *Monitor delivery metrics:* Keep a close eye on your email analytics, particularly failed deliveries and spam rejections. Use tools like SendGrid or Mailgun that provide detailed delivery reports.
2. *Maintain list hygiene:* Regularly clean your mailing list by removing inactive subscribers and hard bounces. This improves your sender reputation and reduces the risk of being flagged as spam.
3. *Implement authentication:* Set up SPF, DKIM, and DMARC records for your domain to verify your identity as a legitimate sender.
4. *Optimize content:* Avoid spam trigger words in your subject lines and content. Use a balanced text-to-image ratio and include an easy unsubscribe option.
5. *Warm up your IP:* If you’re using a new IP address, gradually increase your sending volume to establish a positive reputation with ISPs.
6. *Use feedback loops:* Subscribe to feedback loops offered by major ISPs to receive notifications when recipients mark your emails as spam.
7. *Test before sending:* Use tools like Mail-Tester to check your email’s spam score before sending it to your entire list.

By implementing these practices and closely monitoring your email performance, you can maintain a strong sender reputation, ensure high deliverability rates, and prevent your domain from being blacklisted.

ConclusionWe’ve built an AI-enabled automated email summary system with CI/CD capabilities. This system aggregates content, uses AI to generate summaries, and sends personalized email digests to users. By leveraging Semaphore’s CI/CD pipeline, we ensure consistent, automated deployments.

To maintain and evolve this system:

• *Regularly update* dependencies and security patches.
• *Monitor* performance metrics and user engagement.
• *Continuously refine* your AI prompts for better summaries.
• *Consider adding* more content sources and personalization features.

Remember, while we used Ethereal Email for testing, you’ll want to switch to a production-ready email service provider when you’re ready to send to real users.

Start building your own automated email summary system today by setting up your Semaphore pipeline and integrating AI-powered content summarization.

Additional ResourcesTo further your understanding of the technologies and concepts used in this tutorial, consider exploring these resources:

• Semaphore Documentation: For in-depth information on CI/CD with Semaphore.
• Anthropic’s Claude API Documentation: To learn more about using AI for content generation.
• Node.js Documentation: For comprehensive information on Node.js development.
• Nodemailer Documentation: To dive deeper into email sending with Node.js.

By following this tutorial and exploring these additional resources, you’ll be well-equipped to create and maintain a sophisticated, AI-powered email summary system that keeps your users engaged and informed.

The post Building an AI-Enabled Automated Email Summary System with CI/CD appeared first on Semaphore.

View Details

In terms of creativity, diversity, and evolution pace, generative AI seems boundless. However, its enterprise implementation remains significantly limited. Businesses are uncertain about making the right choices around AI and struggle to familiarize themselves with the potential of this technology amid cutting through the fragmented offerings from major tech companies. In this episode, Arun Gupta, VP and GM for Open Ecosystem at Intel, tells us about the company’s contributions to the open-source community and shares his experience at the Open Platform for Enterprise AI (OPEA) developing open-platform AI solutions for businesses.

Edited transcription

Arun Gupta’s extensive experience driving cultural change toward open source software includes working at Apple (where he built their first open source program office), Amazon, Red Hat, and Sun Microsystems — and later Oracle, after Sun was acquired. Currently, he is the vice president and general manager of Intel’s Open Ecosystem, and consequently, one of the company’s leading voices on everything open source.

Across industries and computer technologies, companies rely on open-source software and expect it to work flawlessly out of the box. “That’s where Intel engineers go out in these open source communities,” says Arun, explaining that Intel’s involvement in open source is about satisfying its diverse client base across verticals and making sure the features are working as intended.

Contributing to the open-source community is as much part of Arun’s job as part of Intel’s DNA. “We have been the largest corporate contributor to Linux since 2007. We are one of the top contributors to Kubernetes. We participate in 300 plus open source projects: Kubernetes, OpenJDK, PyTorch, TensorFlow, GCC, Clang,” he affirms. Arun also points out that Intel’s participation in “700 plus standard bodies and foundations,” benefits the sustainability of the open-source community as much as the company itself.

Foundational leadership in community projectsAside from code itself, Intel contributes to the sustainability of open source projects largely through foundational leadership. The company has recently launched the Unified Acceleration Foundation, which aims to create an open standard for accelerated computing. Intel is also “a premier member” of the LF AI & Data foundation, and holds a seat on the governing board. “The effort is to build and support an open AI and data community and data-driven open source innovation in the AI and data domains,” Arun explains.

From its leadership position, Arun argues, Intel can “strategically shape the direction and initiatives of the foundation […] and technically, strategically, administratively, financially steer the direction to represent our customers’ and developers’ interest.”

Advocating for enterprise AI: The OPEA projectDespite the popularity and growing demand for AI solutions, businesses are still in the early stages of learning how to leverage them. Diving into this new field brings up a lot of inquiries about use cases, security, and the fittest implementation approach. Big tech companies, for their part, are aware of the lack of consensus and unfamiliarity around enterprise AI implementation and want to capitalize on it. “Every hyper scaler is crafting their own bespoke solution and saying, ‘We know it the best’,” says Arun.

To remedy this gap, Arun is currently deeply involved in the Open Platform for Enterprise AI (OPEA) project, an open platform initiative under the LF AI & Data foundation, aimed at simplifying and freeing the development of enterprise AI applications.

As a development framework, OPEA can be used to build gen AI solutions, such as chatbots, providing microservices components like embedding or data storage and the possibility of creating LLMs from your data with improved accuracy using Retrieval Augmented Generation (RAG). As a complementary feature, OPEA offers assessments to evaluate the readiness and trustworthiness of applications and determine at what stage of development they are.

As in the case of AI itself, Arun affirms the project is growing at a marvelous pace: “We launched the project three and a half months ago now. At that time, we had 14 launch partners. We are now up to more than 40 launch partners.” Among these partners are prominent actors such as Canonical, Infosys, Neo4j, Prediction Guard, and Docker, whose interest ranges across different types of AI applications fitting their business use case.

To balance this growth speed and array of contributors, the project has a technical steering committee with members across “a wide range of the community,” says Arun, including a couple from Intel, since “that’s where the largest code contribution is coming in from.” Still, to keep transparency, he keeps his project suggestions and issues in the Github repository, rather than hidden away in “ internal project meetings.” “I just fight the issue on GitHub,” says Arun, “because that’s part of the cultural change that we need to bring, that if the leaders of the project are going to do some sidebar discussions, how would people outside the company understand?”

Regarding deployment, developers looking to get started quickly can package OPEA’s components as Docker containers and publish them in Docker Hub. This containerization strategy forms the backbone of OPEA’s deployment flexibility. “As we are building these microservices, using containers as a base layer for the microservices allows us to run it on any cloud-native platform,” Arun explains. Moreover, OPEA’s deployment options extend far beyond. “You could get this up and running on Kubernetes, for example, your EKS or GKE or AKS, doesn’t matter, pick your favorite flavor of Kubernetes, and we can get it up and running over there,” Arun explains.

The bottom lineOPEA is preparing a hackathon and inciting students and professionals worldwide to participate. If you want to be even more involved, OPEA also has working groups dedicated to different areas such as security, end users, and community. To learn more, visit opea.dev.

To keep updated with the OPEA project and participate in technical discussions first-hand, join OPEA’s mailing list and try the project out for yourself. If you have any questions about where to start, reach out to info@opea.dev.

Likewise, you can contribute by visiting and participating in OPEA’s Github page. Check out the GenAiComps and GenAIExamples repositories for a library of microservices components and a collection of generative AI examples respectively.

Follow Arun on X and Linkedin.

Note: Since this interview was recorded, OPEA has grown to more than 40 partners.

The post Arun Gupta on Democratizing Enterprise AI Development appeared first on Semaphore.

View Details

One of the most challenging tasks in writing reliable, consistent, maintainable, and easy-to-read tests is structuring them correctly. Failing to follow a specific structure in each unit test can lead to flaky or unreliable tests—the nightmare of every QA developer.

Over the years, many approaches to unit testing have emerged, with the Arrange-Act-Assert (AAA, or 3A) pattern standing out as the most popular. This pattern involves structuring your unit tests into three steps:

• Arrange: Set up the test environment.
• Act: Execute the code to test.
• Assert: Verify the results.

In this guide, you will learn what the AAA pattern is, how it works, the benefits it offers, and its role in unit test automation.

Let’s dive in!

The AAA Pattern in Unit TestingThe Arrange-Act-Assert pattern, also known as the AAA or 3A pattern, is a widely recognized approach to structuring tests. It was originally proposed by Bill Wake in 2001 and then mentioned in Kent Beck’s influential book “Test Driven Development: By Example” in 2002.

The AAA pattern promotes clarity by recommending that tests be structured into three distinct phases:

• Arrange everything necessary to perform the test.
• Act on the target code to be tested by executing it.
• Assert the expected outcomes.

The AAA pattern enhances readability and maintainability, closely mirroring the Given-When-Then structure developed by Daniel Terhorst-North and Chris Matts as part of BDD (Behavior-Driven Development). Today, nearly all modern unit testing tools with BDD syntax encourage using the AAA pattern.

Time to explore the three phases it consists of!

ArrangeThe Arrange step is where you prepare everything needed for executing the test to make sure it produces accurate results. This involves initializing objects, configuring dependencies, and setting up the environment required for the test scenario.

Examples of operations you may need to perform in this phase include:

• Creating instances of the classes to be tested
• Initializing global or local variables with specific values
• Setting up mock objects to simulate external services
• Populating a sample database with test data
• Configuring specific settings and configurations

By thoroughly arranging the test context and state, you ensure that the actions and assertions that follow occur in a controlled and predictable environment. This approach enhances test reliability and reduces the likelihood of flaky tests.

ActThe Act step is where you execute the specific functionality you want to test. This phase involves interacting with the SUT (System Under Test) by calling a method or executing a function that you arranged in the previous step.

In most unit tests, this phase corresponds to calling a function or method of an object/class to perform a specific operation. The goal is to execute the action that will generate the outcome you intend to verify in the next step.

To keep the test easy to read and maintain, this action should be concise and focused. A single line of code where you call the method or function to test is generally enough. This simplicity makes it easier to observe and understand the effect of the specific code under test.

AssertThe Assert step is where you verify that the outcome of the unit test matches your expectations. This phase entails checking the results produced by the Act phase against the expected values to confirm that the SUT behaves as desired.

That typically involves using assertion methods to compare the actual results with the expected ones. Examples of assertions include:

• Verifying that a method returns the correct value
• Checking that an object’s state has changed as anticipated
• Guaranteeing that an exception is thrown under certain conditions

To maintain a test’s readability and effectiveness, the Assert step should be clear and specific. This is why it is recommended to have a single assertion—or a small set of related assertions—per unit test.

Example of a Unit Test Written With the AAA PatternTo better understand how the Arrange-Act-Assert pattern works, take a look at the sample Mocha unit test below organized according to the AAA pattern:

import { expect } from "chai"import { MathUtils } from "src/utils/MathUtils";describe("MathUtils Tests", function () { describe("#getFibonacciNumber()", function () { it("should return the correct Fibonacci number", function () { // Arrange: Initialize the class to test const mathUtils = new MathUtils() // Act: Test the method of a class const result = mathUtils.getFibonacciNumber(6) // Assert: Verify that the method produces the expected outcome expect(result).to.equal(8) }) })}) Now, let’s break down the code to highlight which lines correspond to each of the three phases!

For more complex examples, check this and this Mocha tests on GitHub.

Arrange const mathUtils = new MathUtils() Here, you set up the test by creating an instance of the MathUtils class.

In this example, the Arrange step occurs directly within the it() function. However, when the test requires a more complex setup like a sample database, the phase is often handled in beforeAll() or beforeEach() hooks.

Note that imports and the describe() functions are not considered part of the Arrange phase, which typically involves only the first lines of it() functions and/or the code in beforeAll() and beforeEach() hooks.

Act const result = mathUtils.getFibonacciNumber(6) In this step, you use the mathUtility object you initialized earlier to execute the getFibonacciNumber() method with a given input.

Note: In this example, the input is simple and can be passed directly to the function under test. When dealing with more complex or multiple inputs, you should assign them to dedicated variables in the Arrange step and then use those variables in the Actphase.

Assert expect(result).to.equal(8) Finally, you use the expect() method from Chai to verify that the result from the getFibonacciNumber() method matches the expected output.

Note: In this case, the output is simple and can be asserted directly. For more complex outputs, you should assign them to dedicated variables in the Arrange step and then use them in the Assert step.

7 Good Reasons to Use the AAA PatternExplore the main benefits of adopting the AAA unit testing pattern.

Programming Language and Testing Framework AgnosticLike other software design patterns, the Arrange-Act-Assert pattern is not tied to any specific programming language or tool. That means you can apply it in any testing framework, including Mocha and Jest in JavaScript, JUnit in Java, and pytest in Python.

This consistency allows the development team to use a unified approach to unit testing across different projects, regardless of the testing technology in use.

Better Code OrganizationThe AAA pattern promotes better code organization by encouraging you to separate unit tests into three distinct phases. This fixed structure helps maintain a consistent format across tests referred to different sections of your code.

By isolating the setup, execution, and verification stages, the pattern reduces clutter and confusion. This ensures that each test is focused and well-structured, resulting in more maintainable test code.

Makes Unit Tests Easier to UnderstandThe direct consequences of more organized code are improved readability and enhanced clarity. A unit test written using the Arrange-Act-Assert pattern is inherently straightforward, as it follows three neat steps.

You can easily understand tests following the pattern, even if you are unfamiliar with the specific testing tool or technology used. That is especially true when the chosen testing technology comes with an intuitive assertion API.

Encourages Test-Driven DevelopmentThe AAA unit testing pattern supports TDD (Test-Driven Development) by imposing an explicit structure for your tests. Each unit test that follows the pattern must consist of three steps, regardless of the code being tested.

As a result, you can define the expected behavior before the code is actually implemented. That is in line with TDD principles, where tests are created first to specify the desired functionality. This guides development so that the code meets the specified requirements.

Supports RefactoringThe Arrange-Act-Assert pattern supports refactoring by isolating changes to specific phases of the test, without affecting the overall test structure.

For example, if you modify how a method in a class accepts inputs, you only need to update the Act step. As long as the class uses the same logic to be instantiated and the method produces the same output, the Arrange and Assert phases can remain unchanged.

This allows you to refactor code confidently, knowing that your tests will require minimal updates.

Leads to Separations of ConcernsThe AAA approach to unit testing fosters separation of concerns by ensuring that each of the three phases—Arrange, Act, and Assert—remains focused on its specific task. This clear division makes tests more modular and secure, as each step is responsible for a distinct part of the test process. By keeping setup, execution, and verification separate, you also reduce overlap in test logic.

Respect Testing Best PracticesOver the years, the Arrange-Act-Assert pattern has become a de facto standard in the industry. It is currently mentioned and recommended in several best practices guides, including:

• “JavaScript Testing Best Practices” on GitHub
• “Node.js Integration Test Best Practices” on GitHub
• “Unit Test Basics” in Visual Studio documentation
• “How To Write a Test” guide on Cypress’ blog
• Pytest’s documentation

The Role of AAA in Test AutomationThe structured approach of the Arrange-Act-Assert pattern plays a key role in effective unit test automation. Particularly, the Assert step makes it easier to understand test results when executing them in CI/CD pipelines.

In the AAA unit test pattern, each test ends with the Assert phase, which typically includes a single or a few specific assertions. That makes the results of a test easier to evaluate consistently and objectively, even through logs.

For instance, consider the test results image from Mocha’s documentation:

From Mocha’s results, it is evident that the “#indexOf()” test suite in the “Array” collection contains two unit tests. The first fails, while the second passes.

By examining the failing unit test, which follows the AAA pattern, you can quickly pinpoint where to intervene to correct the issue:

describe("Array", function () { describe("#indexOf()", function () { it("should return -1 when not present", function () { // arrange const inputArray = [1, 2, 3] // act const result = inputArray.indexOf(4) // assert expect(result).to.equal(1) }) }) // other test...}) The Arrange step does not contain errors. Similarly, the Act step is implemented as intended. The problem lies in the Assert phase, where the - sign before 1 is missing.

The test asserts the output against a wrong value and fails accordingly. You can fix that with:

expect(result).to.equal(-1) The AAA pattern leads to tests that are straightforward for developers and QA engineers to interpret, even starting from result logs alone. This is essential for swiftly identifying and addressing issues, thereby speeding up the process of fixing broken deployments.

As demonstrated, the AAA pattern facilitates unit test automation. To learn more about this practice, check out our guide on the six principles of test automation.

ConclusionIn this guide, you learned about the Arrange-Act-Assert pattern and what a unit test structured using this approach looks like. You explored each step that the pattern recommends for designing readable and maintainable tests, and the benefits it brings to your testing operations. Backed by the community and widely regarded as a de facto standard, the AAA pattern also proves to be a valuable tool in unit test automation.

The post The Arrange, Act, and Assert (AAA) Pattern in Unit Test Automation appeared first on Semaphore.

View Details

Following on the Role Based Access Control (RBAC) improvements announced in April, we’re rolling out a new feature for Scaleup plans: Custom Roles.

Now, in addition to the default roles, you can create your own with tailored permissions for organizations and projects — giving you the control to decide exactly what actions users can take.

What problem do custom roles solve?Semaphore uses a RBAC model to control what resources and actions users can take. This feature included three default roles for organizations and projects.

While these default roles are enough for many, they cannot fit everyone’s needs. Let’s say you wish to give the finance team access to the Billing page to view spending. Before custom roles, the only way to allow access was using the Admin role, which gave unfettered access to almost everything inside the organization. This is far from ideal from a security perspective.

With the addition of custom roles, you can now create roles to fit the user’s needs. For example, you can create a Finance role that can only view the Billing page and nothing else.

Other use cases for custom roles we envision are:

• prevent members from viewing or changing secrets
• allow the compliance team view-only access to the audit logs, members, and group lists
• provide view-only access to a third party for demo purposes
• allow the operation team to safely manage infrastructure and self-hosted agents
• prevent contributors from starting or attaching SSH sessions on CI/CD jobs

So, what are custom roles?Custom roles is a new Scaleup plan feature that allows you to create an unlimited number of new roles. You can then assign the roles to users or groups to precisely manage what resources they can access and actions they can take.

Same as with default roles, there are two kinds of custom roles:

• Organization roles: Grant actions on the organization such as viewing the dashboards or the audit logs, managing secrets or notifications, and configuring self-hosted agents or pre-flight checks. You’ll find view and manage permissions for most resources on Semaphore.
• Project roles: Grant actions on projects such as view or manage artifacts, perform job debugging, view secrets, or run tasks, to name a few.

Custom roles improve your security posture in many ways:

• Flexibility: you can tailor permissions to your specific needs.
• Security: you can follow the principle of least privilege and lock down user access.
• Scalability: you can create or modify roles to adapt as your organization grows.
• Compliance: custom roles let you meet regulatory requirements.

How to create custom rolesInitially, only the organization Owners can create custom roles. You can, however, create a role that can view or manage custom roles.

Organization rolesOrganization roles determine what actions user can take in the organization.

To create an organization-level custom role:

1. Open the Organization Settings menu
2. Select Roles

3. On the Organization Roles section, press New Role

4. Give a name a description to the new role

5. Enable the permissions for this role. Use the search box to narrow down options

6. Optionally, you can grant a project role for all projects in the organization. For example, the role might have view-only access to every project.

7. Press Save changes

The default roles cannot be modified, but you can see their permissions by pressing the eye button.

Once the role is created, go to People in the organization settings menu, press Change Role, and select the new role. You can only assign one role to an individual or group.

Project rolesProject roles determine what actions users can take within a single project.

To create a project role:

1. Open the Organization Settings menu
2. Select Roles

3. On the Project Roles section, press New Role

4. Give a name a description to the new role

5. Enable the permissions allowed to the role. Use the search box to narrow down options

6. Press Save changes

You can grant project roles by going to the project and selecting the People tab. Press Change Role, and select the new role. You can only assign one role to an individual or group.

ma

FAQWhen will the feature be available?The feature will be generally available on September 30th, 2024.

Help! I can’t find custom rolesThere might be a few reasons why you can’t find the New Role button in the roles setting page:

• You might not be the Owner of the organization
• Your organization might not be subscribed to the Scaleup plan

If none of these conditions apply, please get in touch with support@semaphoreci.com so we can help you resolve the issue.

How can I view the available permissions?Open the organization menu and select Settings. Go to the Roles section and press the eye or pencil button next to the role.

You will see the list of all available permissions. The actions enabled for the role are the ones checked.

Why I can’t change default roles?Default roles permissions cannot be changed. This may change in the future, but for now, default role settings are read-only.

How does this impact the GitHub permission integration?The GitHub permission integration is not affected in any way. Users may still gain project-level permissions via their roles in related repositories.

For more information, check the GitHub permissions page.

ConclusionCustom roles give you a new layer of security governance. It is now possible to lock down user permissions very tightly to comply with regulatory requirements. We hope this feature makes user management easier and more secure.

Thank you for reading and happy building!

The post Custom Roles Are Here appeared first on Semaphore.

View Details

In terms of creativity, diversity, and evolution pace, generative AI seems boundless. However, its enterprise implementation remains significantly limited. Businesses are uncertain about making the right choices around AI and struggle to familiarize themselves with the potential of this technology amid cutting through the fragmented offerings from major tech companies. In this episode, Arun Gupta, VP and GM for Open Ecosystem at Intel, tells us about the company’s contributions to the open-source community and shares his experience at the Open Platform for Enterprise AI (OPEA) developing open-platform AI solutions for businesses.

Edited transcription

Arun Gupta’s extensive experience driving cultural change toward open source software includes working at Apple (where he built their first open source program office), Amazon, Red Hat, and Sun Microsystems — and later Oracle, after Sun was acquired. Currently, he is the vice president and general manager of Intel’s Open Ecosystem, and consequently, one of the company’s leading voices on everything open source.

Across industries and computer technologies, companies rely on open-source software and expect it to work flawlessly out of the box. “That’s where Intel engineers go out in these open source communities,” says Arun, explaining that Intel’s involvement in open source is about satisfying its diverse client base across verticals and making sure the features are working as intended.

Contributing to the open-source community is as much part of Arun’s job as part of Intel’s DNA. “We have been the largest corporate contributor to Linux since 2007. We are one of the top contributors to Kubernetes. We participate in 300 plus open source projects: Kubernetes, OpenJDK, PyTorch, TensorFlow, GCC, Clang,” he affirms. Arun also points out that Intel’s participation in “700 plus standard bodies and foundations,” benefits the sustainability of the open-source community as much as the company itself.

Foundational leadership in community projectsAside from code itself, Intel contributes to the sustainability of open source projects largely through foundational leadership. The company has recently launched the Unified Acceleration Foundation, which aims to create an open standard for accelerated computing. Intel is also “a premier member” of the LF AI & Data foundation, and holds a seat on the governing board. “The effort is to build and support an open AI and data community and data-driven open source innovation in the AI and data domains,” Arun explains.

From its leadership position, Arun argues, Intel can “strategically shape the direction and initiatives of the foundation […] and technically, strategically, administratively, financially steer the direction to represent our customers’ and developers’ interest.”

Advocating for enterprise AI: The OPEA projectDespite the popularity and growing demand for AI solutions, businesses are still in the early stages of learning how to leverage them. Diving into this new field brings up a lot of inquiries about use cases, security, and the fittest implementation approach. Big tech companies, for their part, are aware of the lack of consensus and unfamiliarity around enterprise AI implementation and want to capitalize on it. “Every hyper scaler is crafting their own bespoke solution and saying, ‘We know it the best’,” says Arun.

To remedy this gap, Arun is currently deeply involved in the Open Platform for Enterprise AI (OPEA) project, an open platform initiative under the LF AI & Data foundation, aimed at simplifying and freeing the development of enterprise AI applications.

As a development framework, OPEA can be used to build gen AI solutions, such as chatbots, providing microservices components like embedding or data storage and the possibility of creating LLMs from your data with improved accuracy using Retrieval Augmented Generation (RAG). As a complementary feature, OPEA offers assessments to evaluate the readiness and trustworthiness of applications and determine at what stage of development they are.

As in the case of AI itself, Arun affirms the project is growing at a marvelous pace: “We launched the project three and a half months ago now. At that time, we had 14 launch partners. We are now up to more than 40 launch partners.” Among these partners are prominent actors such as Canonical, Infosys, Neo4j, Prediction Guard, and Docker, whose interest ranges across different types of AI applications fitting their business use case.

To balance this growth speed and array of contributors, the project has a technical steering committee with members across “a wide range of the community,” says Arun, including a couple from Intel, since “that’s where the largest code contribution is coming in from.” Still, to keep transparency, he keeps his project suggestions and issues in the Github repository, rather than hidden away in “ internal project meetings.” “I just fight the issue on GitHub,” says Arun, “because that’s part of the cultural change that we need to bring, that if the leaders of the project are going to do some sidebar discussions, how would people outside the company understand?”

Regarding deployment, developers looking to get started quickly can package OPEA’s components as Docker containers and publish them in Docker Hub. This containerization strategy forms the backbone of OPEA’s deployment flexibility. “As we are building these microservices, using containers as a base layer for the microservices allows us to run it on any cloud-native platform,” Arun explains. Moreover, OPEA’s deployment options extend far beyond. “You could get this up and running on Kubernetes, for example, your EKS or GKE or AKS, doesn’t matter, pick your favorite flavor of Kubernetes, and we can get it up and running over there,” Arun explains.

The bottom lineOPEA is preparing a hackathon and inciting students and professionals worldwide to participate. If you want to be even more involved, OPEA also has working groups dedicated to different areas such as security, end users, and community. To learn more, visit opea.dev.

To keep updated with the OPEA project and participate in technical discussions first-hand, join OPEA’s mailing list and try the project out for yourself. If you have any questions about where to start, reach out to info@opea.dev.

Likewise, you can contribute by visiting and participating in OPEA’s Github page. Check out the GenAiComps and GenAIExamples repositories for a library of microservices components and a collection of generative AI examples respectively.

Follow Arun on X and Linkedin.

Note: Since this interview was recorded, OPEA has grown to more than 40 partners.

The post Arun Gupta on Democratizing Enterprise AI Development appeared first on Semaphore.

View Details

Application programming interfaces (APIs) are the backbone of modern web development, enabling communication between online applications. Not surprisingly, API backends have quickly become the prime target of cyberattacks. Protecting against these threats through robust API security best practices is vital to safeguard your backend applications.

In this article, you will dig into the importance of API threat protection and explore some key techniques for securing your backends.

Let’s dive in!

What Is API Security?API security is the process of protecting APIs against common web attacks, such as data breaches, unauthorized access, and code injection. It focuses on all the practices, strategies, and measures implemented to shield API endpoints and the data they handle from threats and vulnerabilities.

That concept applies to any backend technology, whether it is Node.js, Spring Boot, or Django, and to any types of endpoints, including RESTful, GraphQL, or others. Check out our guide for more guidance on securing Node.js applications in production.

As the great majority of modern web applications rely on APIs, ensuring API security has become a fundamental aspect of the web development process.

Why API Security Is So ImportantAPI security is essential because the web is inherently insecure. As technology evolves, so do the strategies and approaches to exploit it. Considering how advanced malicious bots have become, the number of threats to backend servers has increased significantly in recent years. According to a report from Imperva, nearly one-third of all Internet traffic comes from bad bots. That is alarming!

Common attacks like SQL injection, denial of service, and man-in-the-middle pose serious risks to your API endpoints. If successful, these attacks can result in severe consequences, including:

• Data leaks: User data and privacy breaches that can lead to legal and regulatory penalties for failing to protect sensitive information. For instance, severe GDPR violations can lead to fines of up to 4% of your global revenue.
• Reduced service availability: Disruptions and downtime can lead to financial losses, user dissatisfaction, and legal issues in case of SLA breaches.
• Loss of reputation: Security breaches tarnish your online reputation, triggering negative opinions in the community.

As you can tell, securing a backend is not merely a precautionary measure. Neglecting security can have tangible, detrimental effects on the future of your application and business.

Top 15 API Security Best PracticesNow that you know what API protection is and why it is critical, protect against attacks by applying the API security best practices outlined below

1. Encrypt Communication via HTTPSBy making sure that your backend is accessible only via HTTPS, you can enhance the confidentiality of data exchanged between clients and your server. This is because HTTPS uses the TLS protocol to create encrypted communication channels, protecting sensitive information like session tokens, passwords, and PII data from eavesdropping and tampering.

Additionally, use Secure cookies as part of this HTTPS-only policy. These special HTTP cookies are sent back to the server by the browser only during HTTPS requests, adding an extra layer of security by ensuring they are transmitted exclusively over encrypted connections.

Investing the time to obtain a legitimate SSL/TLS certificate is also important for enhancing the credibility of your backend. While free options like Let’s Encrypt are available, acquiring a certificate from a trusted CA (Certificate Authority) offers additional validation to users.

1. Authenticate Your EndpointsAuthenticating the endpoints in your backend is essential to ensure that only trusted users can access them. Popular API authentication methods include:

2. JWT: JSON Web Tokens provide stateless token-based authentication, allowing secure information exchange between parties.

3. Basic auth: Using a Base64-encoded username and password in the Authorization header for authenticating server-to-server communication.
4. API keys: Strings used as unique identifiers to authenticate requests. This technique is commonly adopted in online services providing public endpoints. Effective strategies for API key management involve regularly rotating keys, implementing usage quotas, and monitoring key activity to detect suspicious behavior. Users should also be educated to set API keys only in environment variables and avoid publicly exposing them.
5. Client SSL certificates: A process where the client and server validate each other’s identities by exchanging SSL certificates. If the validation is successful, they proceed to establish a secure connection. This mutual authentication approach is commonly used in enterprise environments where high security is required for client interactions.

As part of your authentication policies, you also want to enforce strong authentication policies. This means inviting users to set strong passwords and/or rely on MFA (Multi-Factor Authentication) and SSO (Single Sign-On).

1. Hide Your API Backends Behind a GatewayYour microservice architecture likely consists of several API endpoints spread across different servers. To protect them all consistently, you should place them behind an API gateway.

API gateways centralize incoming traffic, routing client requests to the correct endpoint on the appropriate server. This means exposing your endpoints only through the API gateway, where you can centralize security measures.

This approach is especially important considering that API security best practices continuously evolve. So, threat protection measures require a lot of changes over time. By centralizing security in an API gateway, maintenance becomes easier and you do not have to keep updating single endpoints.

Also, do not forget that the most secure API backend is one that is not even exposed to the public Internet but is only accessible through a private network. Implementing this involves placing your APIs behind a bastion host or a VPN, giving access only to authorized users who can tunnel into the private network.

1. Enforce Fine-Grained Access Control StrategiesFine-grained access control (FGAC) is an authorization strategy to allow or deny access to data and resources based on multiple aspects. In particular, it involves granting specific permissions to users based on operations they need to perform on selected resources.

FGAC follows the principle of least privilege, making sure that each user has the minimum required permissions to do what they need to do. This reduces the risk of unauthorized operations and data breaches.

Popular fine-grained access control strategies are:

• RBAC (Role-Based Access Control): Assigning permissions to users based on roles (e.g., admins, editors, visitors, etc.)

• ABAC (Attribute-Based Access Control): Granting access based on user attributes and context (e.g., device, time, location, etc.).

• Implement API Rate LimitingAPI rate limiting is a protection technique that restricts the number of requests a user can make within a specific timeframe. This involves setting constraints on incoming requests to mitigate threats such as DDoS attacks, brute force attempts, and scraping bots.

Two common techniques to implement rate limiting are:

• Block requests: Automatically block any requests that exceed the defined rate limits, returning a 429 Too Many Requestserror.
• Delay requests: Introduce a delay before responding to requests that exceed the limits, slowing down the requester.

By limiting the request rate, you can avoid server overloads and ensure fair usage to all users.

1. Limit the Size of Incoming RequestsMost backend servers, frameworks, and libraries have a default request body size limit set to a few megabytes. This can become an issue during request flooding attacks. The problem is that parsing incoming requests takes time and resources, which increases as the request body becomes larger.

To prevent your server from being overwhelmed by large requests, set a size limit for incoming requests. This setting can typically be configured at both the server level (e.g., using the LimitRequestBody directive in Apache) and the application level (e.g., by limiting the size of the incoming request body).

1. Use Standard Error MessagesEach HTTP error code is associated with a standard name, such as “Not Found” for 404. Since these names are concise and not very descriptive, you may be tempted to include longer, more detailed messages in your error responses. Longer messages will help the client better handle the error, but they may also reveal useful information to malicious users.

As a rule of thumb, avoid detailed error responses including stack traces or technical reasons behind errors. This information does not add real value to the client and will only make it easier for attackers to exploit vulnerabilities in your backend. For enforcing API security, opt instead for generic and standardized error messages.

1. Remove the Default Framework HeadersPopular backend technologies often set default HTTP headers in their responses. For example, Express.js and others automatically set the X-Powered-By header with the framework’s name and sometimes even its version.

These default headers expose the underlying technology stack, helping attackers target known vulnerabilities. As a fundamental API threat protection measure, remove these headers to reduce your backend’s attack surface and reduce potential risks.

1. Validate and Sanitize User InputInput validation ensures that the data received by endpoints adheres to specific formats and respects expected constraints. This API security best practice reduces errors in the business logic of your backend and mitigates XSS (Cross-Site Scripting) attacks.

Meanwhile, input sanitization focuses on removing potentially harmful characters from input data. That protects against popular code injection attacks like SQL injection.

1. Set the Security HTTP HeadersSecurity HTTP headers are a set of HTTP headers designed to add an additional layer of protection against common web vulnerabilities. These headers include Referrer-Policy, Content-Security-Policy, X-Content-Type-Options, and others. You can check whether your backend sets them correctly at the Security Headers site.

For example, defining a CSP policy through the Content-Security-Policy allows you to specify which trusted sources the browser can load resources from. That mitigates XSS attacks and other common threats.

1. Integrate Logging and Application MonitoringWhen something goes wrong, you need to understand when, what, why, and how it happened. This is exactly what logging is all about. By tracking your backend activities in log text files, you can reconstruct events in case of errors to address them more easily.

Similarly, application performance monitoring (APM) involves tracking the performance and health of software. Integrate an APM library into your backend, and this will start sending useful data in the background using standard protocols like OpenTelemetry. That data can then be used to get real-time insights and set alerts for unusual behavior, helping immediately identify suspicious activities and security breaches.

1. Regularly Perform Audits, Penetration Tests, and Stress TestsA code audit for security involves examining server configurations and the codebase of your backend to ensure compliance with security standards. Instead, penetration tests are more proactive and involve simulating real attacks to identify potential entry points for attackers. Finally, stress tests evaluate API performance under high loads, ensuring resilience against denial-of-service attacks and automated bots.

Regularly conducting code audits, penetration tests, and stress tests helps discover weaknesses in your backend and address them before malicious attackers can take advantage of them.

1. Use Security Linters, SCA, and SAST ToolsSecurity linter plugins, such as eslint-plugin-security, analyze your code for known vulnerabilities and adherence to API security best practices. In detail, they flag potential issues like unsafe regular expressions or dangerous import instructions.

SCA (Software Composition Analysis) tools evaluate the dependencies used in your application for known vulnerabilities. They identify outdated or insecure components and recommend updates or replacements to keep your software stack secure. Tools like Snyk and Black Duck are popular for performing these analyses. In general, you should always keep your third-party libraries up-to-date, relying on such a service to alert you when known security issues emerge.

SAST (Static Application Security Testing) tools perform comprehensive scans of your codebase to identify security flaws in the logic or architecture of your backend. Examples of SAST tools are SonarQube and Lacework.

By integrating such security tools into your development workflow, you can proactively prevent security risks.

1. Use API FirewallsAPI firewalls monitor and filter both incoming and outgoing traffic for malicious activity. When it comes to backend protection, all-in-one web application firewall (WAF) solutions like Cloudflare, AWS WAF, and Akamai have become particularly popular.

WAFs provide comprehensive security by blocking known threats, preventing unauthorized access, and mitigating known attacks such as SQL injection and cross-site scripting. In addition, they generally offer features like rate limiting and bot protection to further enhance the security of your APIs.

1. Follow Established Security StandardsAPI security is a cat-and-mouse game, with new threats emerging daily and constant updates to security systems to protect against them. This is why staying abreast of the latest developments is imperative!

Adhering to trusted sources like OWASP offers invaluable guidance, helping you keep your backend robust and up-to-date in the face of evolving threats. Specifically, the OWASP Top Ten project provides crucial insights into the most prevalent web threats.

ConclusionIn this guide, you learned how to safeguard backend applications through the adoption of API security best practices. As a developer, it is your responsibility to maintain application functionality, safeguard your users, and prevent data breaches. By incorporating these techniques, you can bolster the resilience of your backend against common web attacks, effectively mitigating risks and safeguarding the security of your application.

The post API Security Best Practices: Protecting Your Data in Transit appeared first on Semaphore.

View Details

Assistive coding is one of the most powerful ways to apply large language models (LLMs). A well-trained model integrated into your development environment can supercharge your productivity.

Hosted models as-a-service have become cheaper and more effective as their adoption accelerates. However, there are cases in which you may want to run your own:

• Privacy: Relying on hosted APIs requires sending your data to a third party to populate context.
• Cost: Iterating often with a large codebase can incur charges quickly and discourage experimenting with different prompts.
• New developments: Models with significant new capabilities appear on an almost weekly (or daily!) basis. Running your own open models can keep you ahead of the curve.

This article will walk you through how to leverage the right self-hosted LLM and integrate it into your environment.

Doing it Yourself: An OverviewThe most popular and effective commercial LLM coding assistant services pair highly-trained models with powerful hardware to deliver accurate suggestions quickly. To build your own, you’ll need the same: a refined model well-suited for coding tasks, compute hardware with sufficient power to generate tokens (or suggestion output) quickly, and an editor or IDE integration to bring it all together.

In this guide we’ll use Ollama to easily run our desired models and assume that you have a functional ollama service API available for use. If not, many resources are available to help you get started with Ollama. Consider using the official quick start documentation as a starting point.

The primary setup considerations to bear in mind in the context of this article are:

• More capable models need significant resources. If you intend to run models beyond 40 billion parameters, you likely need powerful local hardware (or optimized cloud resources).
• Choosing the right model is as much an art as it is a science. This tutorial will help explore models and their capabilities, but experimenting with different models against your use case will help you select the most effective strategy.

We’ll look at editor integrations for VSCode, Vim, and Emacs — though most editor ecosystems provide some method of integration with LLM APIs. When evaluating an LLM integration, API compatibility against Ollama is the primary requirement. The models themselves can be swapped out from underneath the Ollama service.

To ensure a fair comparison across both models and editor integrations, we’ll compare models with similarly-sized parameters, then integrations, and finally demonstrate the two paired together.

A Brief Tour of Coding ModelsBefore diving into code-optimized LLMs, we should establish a baseline prompt to gauge their effectiveness against a uniform set of requirements.

Consider the following example:

Write a Python script that uses the click library to accept a URL argument that the script will POST input to with the request body populated from stdin and then pretty print the resulting response body. Print a friendly message if the response returns an HTTP error status code. Do not provide commentary or any other explanatory text, return code only.

This prompt judges a model’s ability to:

• Use standard libraries (to read input from standard input)
• Use third-party modules (with the click library)
• Write functions that accept and use arguments
• Perform network HTTP operations
• Present user-readable output
• Implement error handling

To follow along, the only prerequisite is a functional Ollama server accessible from your own machine with an ollama run command. If you’re running Ollama on your local machine, then ollama commands should default to localhost as the API endpoint. If you’re running Ollama on a cloud resource or other remote machine, then invoke your commands in the form of:

OLLAMA_HOST=ollama-host:11434 ollama run Where ollama-host is the destination hostname or IP address of the remote server. The examples that follow are results derived from entering our test instructions at the > prompt from the ollama run command.

With this criteria in hand, let’s explore!

codeqwencodeqwen is a coding-optimized model with a parameter size small enough at 7B parameters that it can fit into the available memory space of consumer-grade GPUs. Given our baseline Python program prompt, here’s an example response from codeqwen:

import clickimport requestsimport json@click.command()@click.argument('url')def post_request(url): try: response = requests.post(url, data=input()) response.raise_for_status() # Raise an HTTPError for 4xx/5xx status codes print(json.dumps(json.loads(response.text), indent=2)) # Pretty print the response body except requests.HTTPError as http_err: print('HTTP error occurred:', http_err) # Print HTTP error message except Exception as err: print('An error occurred:', err) # Print other errorsif __name__ == '__main__': post_request() Despite its relatively lightweight footprint, codeqwen does an excellent job at fulfilling our prompt, honoring our requirements with good patterns:

• It leverages the requests library without any hints to do so.
• Error handling catches both HTTP errors and other exceptions.
• The standard json library pretty-prints responses (with indent=2).

The source format is suitable for use as a script and meets the requirements defined in our baseline prompt.

deepseek-coderThe deepseek-coder model has sizes similar to those available to codeqwen, ranging from 1B to 33B. Although a v2 deepseek-code model exists, we’ll contrast the 6.7B deepseek-coder model against codeqwen:7B to compare across similar parameter sizes.

Given the same initial prompt, deepseek-coder:6.7B returns the following Python:

import clickimport requestsfrom pprint import pformat@click.command()@click.argument('url')def post_request(url): try: resp = requests.post(url, data=click.getchar()) if 400 <= resp.status_code < 600: click.echo("An HTTP error {} occurred.".format(resp.status_code), err=True) else: click.echo(pformat(resp.json())) except requests.exceptions.RequestException as e: click.echo('Error: {}'.format(e), err=True)if __name__ == '__main__': post_request() Like codeqwen:7B, deepseek-coder:6.7B correctly uses the chosen libraries and infers the use of requests as an HTTP client. Some noteworthy differences include:

• Using the native pprint library to pretty-print JSON parsed from the .json() requests method.
• Manually checking that the response code falls within a successful range. This is slightly less ergonomic than codeqwen’s .raise_for_status() approach.
• The click.getchar() function only accepts one character, and does not read from standard input.

This solution is mostly functional but misses subtle success criteria from our original requirements. For example, it cannot accept an entire payload from standard input rather than one character from an interactive prompt.

codellamacodellama is a Llama 2-based model from Meta tuned for coding and available in many different parameter sizes including 7B, 13B, 34B, and 70B. We’ll use the 7B model to compare model capabilities as with codeqwen and deepseek-coder.

import clickimport requestsfrom rich.console import Console@click.command()@click.argument("url")def main(url: str): console = Console() try: response = requests.post(url, data=console.input()) if response.status_code == 200: console.print(response.json()) else: console.print("Error: {}".format(response.status_code)) except Exception as e: console.print("An error occurred: {}".format(e)) Once again, the differences between this solution and the previous attempts are interesting to explore:

• The use of type annotations in main() is an optional but generally good practice to improve the readability of the code.
• The rich library accepts input and pretty-prints the response which is an effective way to repurpose the library to serve two purposes. The console.input() method also properly accepts input via either stdin or command line input.
• Without a call to main() we’ll need to either invoke it manually or wrap it with a standard conditional check against __name__ == "main".

While missing the trailing __name__ check at the end of the generated code, the use of features like type annotations and additional libraries suggests that codellama may be well-suited for code that follows best practices. Larger parameter sizes also offer the potential for more comprehensive code completion.

llama3.1Although not specifically tuned for coding, the llama3.1 model is a recent addition to the family of open-source models. With parameter weights ranging from 8B to 405B, it can compete with even the most capable commercial LLM offerings.

Queried against our baseline prompt, the 8B model returns the following:

import clickimport jsonfrom requests import postfrom sys import stdin@click.command()@click.argument('url')def main(url): """POST input to URL and print response""" req = post(url) print(json.dumps(req.json(), indent=4)) if req.status_code < 400: return click.echo(f"Error {req.status_code}: {req.reason}", err=True)if __name__ == '__main__': main() Despite being a generalized model, llama3.1 performs relatively well against models trained specifically for coding tasks:

• The requests library is the only other additional dependency besides click.
• Unlike every other generated response, this function includes a document string.
• The code snippet is complete with a trailing conditional to run as a script when invoked from the command line.

However, the generated code has a glaring bug: it never accepts any input! Despite importing stdin, the call to requests.post never includes any request body. While the solution is well-formatted and runs, it misses our requirement for payload input entirely.

In an environment with sufficient computing power, llama3.1 may still be worth exploring. Models with more parameters are more capable than most of the other models we’ve explored so far.

Editor IntegrationIntegrating your chosen model into an editor is a key step to making full use of an LLM. Both chat-based assistive coding and autocomplete-style generation can significantly improve their utility.

The Ollama API abstraction layer provides a uniform access method to different coding models, which makes editor integration simpler. Even across different coding assistant extensions or packages, as long as each is compatible with the Ollama API, the backend can remain the same without any significant friction.

Let’s look at integrations that can present a convenient interface in a variety of editors.

VSCodeAmong the Ollama-compatible extensions available on the VSCode Extension Marketplace, we’ll use Ollama Autocoder as an illustrative case for integrating an Ollama model into VSCode.

After installing the extension, open the settings for the extension:

Scroll to the Endpoint setting and change it to the Ollama API endpoint for your running service. For example, in this screenshot, replace ollama-host with either your Ollama endpoint IP address or hostname:

After making these changes, you’re ready to try Ollama-powered autocompletion!

Open a Python source code file and begin with an appropriate script shebang like #!/usr/bin/env python. Follow it with a comment including our test prompt, and position the cursor at the end of the file. The extension will pick up at the cursor location with text preceding as context to the model. Use the command palette (Ctrl-Shift-P) to execute the command Autocomplete with Ollama. The extension should stream generated code:

The preceding recording is generating code from the codeqwen model running on a legacy NVidia 1070 GTX GPU.

EmacsPackages like gptel and org-ai offer integration with LLM-based APIs, but this example will use ellama as a simple example of code completion with Ollama.

The following elisp snippet uses the use-package macro to install and configure the ellama package and set its provider backend to an ollama API at http://ollama-host:14434:

(use-package ellama :config (require 'llm-ollama) (setq ellama-provider'(make-llm-ollama :host "ollama-host" :port 11434 :chat-model "codeqwen" :embedding-model "codeqwen"))) Reload Emacs after placing the code snippet into ~/.emacs.d/init.el or evaluate the use-package form. Position the point in a source code file where you’d like to generate code and invoke M-x ellama-code-complete. The following recording demonstrates this in a Python source file preceded by our baseline prompt:

As with the VSCode extension, ellama will rely on the Ollama API endpoint to generate code based on the editor context. Completion input streams into the open buffer until it finishes.

NeovimThe Neovim plugin model.nvim offers the ability to integrate with APIs like Ollama and provide streaming completion as with the integrations with VSCode and Emacs.

To install and configure model.nvim with the lazy.nvim plugin manager, use the following configuration under the section for require("lazy"). This will also prepare the plugin to use the hosted Ollama API endpoint:

require("lazy").setup({ spec = { "gsuuon/model.nvim", config = function() local ollama = require('model.providers.ollama') require('model').setup({ prompts = { ['ollama:codeqwen'] = { provider = ollama, options = { url = "http://ollama-host:11434", }, params = { model = "codeqwen", }, builder = function(input) return { prompt = '<|im_start|>' .. input .. '<|im_stop|><|im_start|>assistant', stops = { '<|im_stop|>' }, } end, } }, }) end, },}) The ['ollama:codeqwen'] prompt stores the configuration for our ollama prompt. Change the url option to point at your ollama API endpoint, listed in this example as ollama-host.

Start Neovim with nvim and open a new Python source buffer with :e main.py and populate the buffer with the baseline prompt. Move the cursor to a newline and invoke the command :Model ollama:codeqwen. The plugin will stream the response into the buffer to complete the code:

Edit the plugin configuration as necessary to define additional models to use with the :Model command to use other models.

SummaryIn this guide, we have:

• Evaluated a variety of different large language models and their effectiveness at fulfilling coding instructions.
• Showcased editor extensions for VSCode, Emacs, and Neovim that integrate with ollama APIs.
• Combined large language models that fit on consumer-grade hardware with those extensions to stream generated code completion.

To continue exploring self-hosted modules:

• Browse the library of Ollama models.
• Install and use Ollama integrations.
• Explore tools optimized for local hosting like llama.cpp and exo.

The post A Guide to Self-Hosted LLM Coding Assistants appeared first on Semaphore.

View Details

Starting October 1st, 2024, Semaphore will remove the Xcode 14 image from the available machine pool. This change affects all Apple developers who are building their projects with Xcode 14.

Apple has announced that as of April 2024, all iOS and iPadOS apps submitted to the App Store must be built with a minimum of Xcode 15 and the iOS 17 SDK

How do I know if I am impacted by this change?If your organization builds projects on Apple machines, you might be affected. Only projects using the Xcode 14 image will be affected by this change.

You can check what image your projects are using with the Workflow Editor.

To see what image you are using follow these steps:

1. Check the settings for all your pipelines. If the OS Image is “macos-xcode14” you are affected

2. Open all your blocks. If the Override global agent definition is enabled and the OS Image is “macos-xcode14” you are affected.

3. Alternatively, you can search for the text "macos-xcode14" in your pipeline YAML files inside the .semaphore folder. If you find this string, you are affected,

How to prepare for the change?If your organization is affected by the deprecation, we encourage you to switch to the Xcode 15 image before the deprecation date.

You can change the macOS image in the Workflow Editor.

Or edit your pipeline YAML files to replace all instances of "macos-xcode14" with "macos-xcode15". For example:

version: v1.0name: Initial Pipelineagent: machine: type: a1-standard-4 os_image: macos-xcode15blocks: - name: 'Block #1' task: jobs: - name: 'Job #1' commands: - checkout We encourage you to test your workflows with the Xcode 15 image thoroughly before making new releases with the newer image.

Feel free to reach out to us at support@semaphoreci.com if you have any questions or need our assistance. We’ll be happy to help.

The post Xcode 14 Deprecation in Semaphore appeared first on Semaphore.

View Details

Access control is essential for application security. It ensures that only authorized users can access specific resources or perform certain actions. Effective access control prevents unauthorized access, safeguards sensitive data, and maintains application integrity.

In the realm of microservices architectures, managing authorization logic across various services can become a complex task. Cerbos offers a solution by centralizing authorization policies. This simplifies the process because you can make decisions from any part of the application. You can avoid a maintenance burden as your application evolves.

In this tutorial, we will explore Cerbos, a modern, open-source solution, for managing and enforcing access control policies. While popular tools like Okta excel at user authentication and basic access control, applications with complex permission structures require a more granular approach. Cerbos offers a flexible method for defining and applying detailed access control rules. It supports various frameworks and languages, such as Python and Java.

In this article, you will learn:

• How Cerbos works and what benefits it offers.
• How to secure a Spring Boot application using Cerbos policies.
• How to test Cerbos policies in the Semaphore CI pipeline.

PrerequisitesTo ensure smooth learning, this tutorial assumes you have:

• Basic Docker knowledge, including containers and Docker Compose.
• Fundamentals of Spring Boot, such as working with JPA entities and REST APIs.
• An integrated development environment (IDE) like IntelliJ IDEA.

Let’s get started!

Understanding CerbosThe core concept of Cerbos is to define all access control rules in a central location in a human-readable format. When you update these rules, Cerbos applies the changes across all instances serving your application. This way, you don’t need to release any code changes.

Cerbos consists of two main components:

• Cerbos Hub: A managed platform for creating and managing policies collaboratively.
• Cerbos Policy Decision Point (PDP): An open-source engine that enforces these policies.

Benefits of Using Cerbos* Developers can concentrate on building core application features while delegating access rule management to product owners or security teams. * You can deploy Cerbos as a separate microservice. * Cerbos makes authorization logic easier. It replaces complex, hard-coded permission checks with a single call to the Cerbos policy engine.

For example, look at this Java boiler-plate code:

With Cerbos, you can simplify it to the absolute minimum:

What Are Cerbos PoliciesYou can define Cerbos policies in YAML or JSON files. These policies consist of two main parts:

• Resources: Specify an entity within your application that requires protection through access rules. For example, an employee record with personal details, salary, etc.
• Rules: Determine who can access each resource and what actions they can perform. For instance, colleagues from the HR department can access and manage employee records.

Cerbos provides several ways to create policies:

• Cerbos playground: An interactive interface for defining and testing policies.
• Cerbos Hub: A managed platform for policy creation and management.
• Manual Configuration: Defining policies directly in YAML files.

In this tutorial, we will use the Cerbos Playground to define our policies.

Integrating Cerbos into a Spring Boot ApplicationProject ScenarioIn our demo application, we will define access control policies for an employee management system. The system has two primary roles: regular employees and HR personnel. The policies will ensure that regular employees can only view their own profiles. HR personnel will have full control over all employee records.

Prepare the ProjectCreate a new Maven project in your favorite IDE. For example, name it CerbosSpringBootDemo.

Create the Cerbos policiesVisit the Cerbos playground.

Create two custom roles: employee and hr. The resource we will manage is the employee’s profile.

• The employee role has read-only access to the profile.
• The hr role has full access to perform all actions.

Finally, press Generate.

You can then see what the policies look like:

On the left panel, you can see the YAML files that contain the policies.

We are interested in the resource_policies folder. Let’s copy the contents of these files and save them to our local machine.

Create a new folder in the root of your project called cerbos-policies. The name can be different but note that Cerbos will look for a folder called policies when running in the container. Create a sub-folder called testdata. Make sure you have the following structure:

├── cerbos-policies│ ├── profile_test.yml│ ├── profile.yaml│ └── testdata│ ├── principals.yaml│ └── resources.yaml * principals.yaml:

principals: employee#1: id: employee#1 roles: - employee attr: {} hr#2: id: hr#2 roles: - hr attr: {} This file defines the principals (users) for the policy tests, specifying their IDs, roles, and attributes.

• resources.yaml:

resources: profile#1: id: profile#1 kind: profile attr: {} This file defines the resources that will be subject to the policies being tested.

• profile.yaml:

apiVersion: api.cerbos.dev/v1resourcePolicy: resource: profile version: default rules: - actions: - create effect: EFFECT_ALLOW roles: - user - admin - hr - actions: - read effect: EFFECT_ALLOW roles: - user - admin - employee - hr - actions: - update effect: EFFECT_ALLOW roles: - user - admin - hr - actions: - delete effect: EFFECT_ALLOW roles: - admin - hr This policy file specifies the resource as the profile and sets rules that allow an employee with the hr role to perform all actions. It restricts employees with the employee role to read their own profiles only.

• profile_test.yaml:

name: profileTestSuitedescription: Tests for verifying the profile resource policytests:- name: profile actions input: principals: - employee#1 - hr#2 resources: - profile#1 actions: - create - read - update - delete expected: - resource: profile#1 principal: employee#1 actions: create: EFFECT_DENY read: EFFECT_ALLOW update: EFFECT_DENY delete: EFFECT_DENY - resource: profile#1 principal: hr#2 actions: create: EFFECT_ALLOW read: EFFECT_ALLOW update: EFFECT_ALLOW delete: EFFECT_ALLOW This YAML file defines a set of tests for verifying the access control policy related to the profile resource. It specifies the actions that different principals (users with roles) can perform on the resource and what the expected outcomes should be. Note that Cerbos expects the file suffix _test.

Prepare the Infrastructure for Local DevelopmentYou can integrate Cerbos into your stack in two main ways:

• By downloading and installing the binaries.
• Using a docker container.

We will pull the official Docker image and run it via docker-compose. We will also deploy our custom policies to the running Cerbos PDP instance.

Create a docker-compose.yml file in the root of your project with the following content:

version: "2.1"services: my-cerbos-container: container_name: my-cerbos-container image: ghcr.io/cerbos/cerbos:0.34.0 ports: - "3592:3592" - "3593:3593" volumes: - ./cerbos-policies:/policies expose: - "3592" - "3593" command: compile /policies Let’s break down the docker-compose.yml file to understand what each section does:

• image: Specifies the Docker image for this service, which is the Cerbos PDP version 0.34.0 from GitHub Container Registry.
• container_name: Names the container my-cerbos-container for easy identification.
• ports: Maps ports 3592 and 3593 on the host machine to the same ports in the container, allowing external access.
• expose: Indicates that ports 3592 and 3593 should be exposed, making them available for linked services.
• volumes: Mounts the ./cerbos-policies directory from the host machine to the /policies directory in the container, providing the Cerbos PDP access to your policy files.
• command: Executes the Cerbos command compile /policies to run the tests in the profile_test.yaml file.

Create the Spring Boot ApplicationAdd the following dependencies to the parent pom.xml file:

<dependencies> <!-- Spring boot dependencies --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>3.2.3</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> <version>3.2.3</version> </dependency> <dependency> <groupId>jakarta.persistence</groupId> <artifactId>jakarta.persistence-api</artifactId> <version>3.1.0</version> </dependency> <!-- Database --> <dependency> <groupId>com.h2database</groupId> <artifactId>h2</artifactId> <version>2.2.224</version> </dependency> <!-- Cerbos dependencies --> <dependency> <groupId>dev.cerbos</groupId> <artifactId>cerbos-sdk-java</artifactId> <version>0.12.0</version> </dependency> <dependency> <groupId>io.grpc</groupId> <artifactId>grpc-core</artifactId> <version>1.64.0</version> </dependency> <!-- Other dev tools --> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.5.6</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.30</version> </dependency> </dependencies> Below is a summary of each dependency:

• spring-boot-starter-web: Provides the core web functionalities to build a Spring Boot web application.
• spring-boot-starter-data-jpa: Adds support for JPA, enabling easy database interaction with repositories.
• jakarta. persistence-api: Supplies the Jakarta Persistence API which is essential for object-relational mapping and managing relational data in Java applications.
• h2: Includes the H2 database engine, a lightweight in-memory database useful for development, testing, and small-scale applications.
• cerbos-sdk-java: Provides the Cerbos SDK for Java, which facilitates integration with the Cerbos authorization system to handle access control policies.
• grpc-core: Supports the core functionality for gRPC (Google Remote Procedure Call), which Cerbos uses for communication.
• logback-classic: Implements logging capabilities using Logback.
• lombok: Offers annotations to reduce boilerplate code in Java, such as generating getters, setters, and constructors automatically.

Configure the application.yml file:

spring: datasource: url: jdbc:h2:mem:testdb driverClassName: org.h2.Driver h2: console: enabled: true jpa: database-platform: org.hibernate.dialect.H2Dialect hibernate: ddl-auto: create show-sql: truelogging: level: root: warn org.cerbos.demo: debug Key points:

• ddl-auto: create: Configures Hibernate to create the database schema and tables at startup. This is helpful for development and testing but should be changed for production.
• h2:console:enabled:true: Enables the H2 console. You can access it at localhost:8080/h2-console when the application starts.
• show-sql: true: Enables the logging of SQL statements generated by Hibernate.
• logging.level: Reduces the root level output and sets only the necessary org.cerbos.demo package to debug.

Let’s configure the Cerbos client. Create a new class CerbosConfig.java:

@Configurationpublic class CerbosConfig { @Bean public CerbosBlockingClient cerbosClient() throws CerbosClientBuilder.InvalidClientConfigurationException { LoadBalancerRegistry.getDefaultRegistry().register(new io.grpc.internal.PickFirstLoadBalancerProvider()); NameResolverRegistry.getDefaultRegistry().register(new io.grpc.internal.DnsNameResolverProvider()); return new CerbosClientBuilder("localhost:3593").withPlaintext().buildBlockingClient(); }} This configuration sets up and registers a CerbosBlockingClient bean in the Spring application context. This allows the application to connect to the PDP to check access control policies. It uses load balancer and name resolver registrations to ensure the gRPC client can find and connect to the PDP server.

You can find more details and custom options for the CerbosClient in the Cerbos-Java GitHub repository.

Let’s create the model for our application. Create a new class Employee.java that represents the employees table:

@Entity@Table(name = "employees")@Datapublic class Employee { @Id @Column(name = "employee_id" , nullable = false) private Long employeeId; @Column(nullable = false) private String name; @Column(nullable = false, unique = true) private String email; @Column(nullable = false) private Double salary; @Column(nullable = false) private String role;} This entity contains usual employee information.

Create the Profile.java that represents the profiles table:

@Entity@Table(name = "profiles")@Datapublic class Profile { @Id private Long id; @OneToOne(cascade = CascadeType.PERSIST) @JoinColumn(name = "employee_id") private Employee employee;} This entity contains the employee details.

Key points:

@OneToOne(cascade = CascadeType.PERSIST): Defines a one-to-one relationship between the Profile and Employee entities. The cascade = CascadeType.PERSIST attribute means that when you persist a Profile entity, you also persist the associated Employeeentity.

Let’s create the JPA repositories: EmployeeRepository.java and ProfileRepository.java. They will allow access to the entities.

public interface EmployeeRepository extends JpaRepository<Employee, Long> { }

public interface ProfileRepository extends JpaRepository<Profile, Long> { } We will build the backend of the Spring boot application and test the endpoints by sending cURL requests.

Create a ProfileController.java for the REST communication:

@RestController@RequestMapping("/api/profile")@RequiredArgsConstructor@Slf4jpublic class ProfileController { private final EmployeeRepository employeeRepository; private final ProfileRepository profileRepository; private final CerbosBlockingClient cerbosBlockingClient; @GetMapping("/get/{profileId}/{employeeId}") public ResponseEntity<String> getProfile(@PathVariable String profileId, @PathVariable String employeeId) { Profile profile = getProfileOrNotFound(profileId); if (profile == null) { return handleProfileNotFound(); } Employee employee = getEmployeeOrNotFound(employeeId); if (employee == null) { return handleEmployeeNotFound(); } Principal principal = Principal.newInstance(employeeId, employee.getRole()) .withAttribute("id", AttributeValue.stringValue(employeeId)); Resource resource = Resource.newInstance("profile", profileId) .withAttribute("owner", AttributeValue.stringValue(String.valueOf(profile.getEmployee().getEmployeeId()))); if (!isAllowed("read", principal, resource)) { log.debug("Not allowed to read profile"); return handleForbidden(); } return ResponseEntity.ok().body(profile.getEmployee().toString()); } @DeleteMapping("/delete/{profileId}/{employeeId}") public ResponseEntity<String> deleteProfile(@PathVariable String profileId, @PathVariable String employeeId) { Profile profile = getProfileOrNotFound(profileId); if (profile == null) { return handleProfileNotFound(); } Employee employee = getEmployeeOrNotFound(employeeId); if (employee == null) { return handleEmployeeNotFound(); } Principal principal = Principal.newInstance(employeeId, employee.getRole()); Resource resource = Resource.newInstance("profile", profileId); if (!isAllowed("delete", principal, resource)) { log.debug("Not allowed to delete profile"); return handleForbidden(); } try { profileRepository.deleteById(Long.valueOf(profileId)); boolean isDeleted = profileRepository.findById(Long.valueOf(profileId)).isEmpty(); return isDeleted ? ResponseEntity.ok().body("Profile deleted successfully") : ResponseEntity.internalServerError().body("Failed to delete profile"); } catch (Exception e) { return handleInternalServerError(e); } } private boolean isAllowed(String action, Principal principal, Resource resource) { return cerbosBlockingClient.check(principal, resource, action).isAllowed(action); } private Profile getProfileOrNotFound(String profileId) { return profileRepository.findById(Long.parseLong(profileId)).orElse(null); } private Employee getEmployeeOrNotFound(String employeeId) { return employeeRepository.findById(Long.parseLong(employeeId)).orElse(null); } private ResponseEntity<String> handleProfileNotFound() { return ResponseEntity.badRequest().body("Profile not found"); } private ResponseEntity<String> handleEmployeeNotFound() { return ResponseEntity.badRequest().body("Employee not found"); } private ResponseEntity<String> handleForbidden() { return ResponseEntity.status(403).body("Forbidden"); } private ResponseEntity<String> handleInternalServerError(Exception e) { log.error("Error processing request: ", e); return ResponseEntity.internalServerError().body("Error processing request: " + e.getMessage()); }} Key aspects and functionality:

The controller fetches data from the JPA repositories. It performs authorization checks using Cerbos and handles both successful and failed operations.

`GET /api/profile/get/{profileId}/{employeeId}:` * Creates Principal and Resource objects to represent the current user and the profile resource. * Uses cerbosBlockingClient to check if the user is allowed to perform the “read” action on the resource. * Returns 403 Forbidden if the access is denied, otherwise returns the profile details.

`DELETE /api/profile/get/{profileId}/{employeeId}:` * Performs the same steps as above, but this time it checks if the user can delete the profile.

Finally, let’s create the Main.java class:

@SpringBootConfiguration@ComponentScan(basePackages = "org.cerbos.demo")@EnableJpaRepositories@EnableAutoConfiguration@Slf4jpublic class Main { static ConfigurableApplicationContext appCtx; public static void main(String[] args) { var app = new SpringApplication(Main.class); appCtx = app.run(args); } @Bean CommandLineRunner commandLineRunner(EmployeeRepository employeeRepository, ProfileRepository profileRepository) { return args -> { populateDb(employeeRepository, profileRepository); }; } void populateDb(EmployeeRepository employeeRepository, ProfileRepository profileRepository) { Employee employee1 = new Employee(); employee1.setEmployeeId(123L); employee1.setName("John Doe"); employee1.setEmail("john.doe@me.com"); employee1.setRole("employee"); employee1.setSalary(1500.0); Employee employee2 = new Employee(); employee2.setEmployeeId(321L); employee2.setName("Marie Smith"); employee2.setEmail("marie.smith@me.com"); employee2.setRole("employee"); employee2.setSalary(2000.0); Employee hr = new Employee(); hr.setEmployeeId(456L); hr.setName("Andrew Anderson"); hr.setEmail("andrew.anderson@me.com"); hr.setRole("hr"); hr.setSalary(1000.0); employeeRepository.save(employee1); employeeRepository.save(employee2); employeeRepository.save(hr); Profile profile = new Profile(); profile.setId(111L); profile.setEmployee(employee1); profileRepository.save(profile); Profile profile2 = new Profile(); profile2.setId(222L); profile2.setEmployee(employee2); profileRepository.save(profile2); Profile profile3 = new Profile(); profile3.setId(333L); profile3.setEmployee(hr); profileRepository.save(profile3); log.debug("Saved data to db"); }} Here, we created a CommandLineRunner bean that populates the database with sample data when the application starts.

Testing the ApplicationI have the following folder structure so far:

├── cerbos-policies│ ├── profile_test.yml│ ├── profile.yaml│ └── testdata│ ├── principals.yaml│ └── resources.yaml├── docker-compose.yml├── pom.xml├── src│ ├── main│ │ ├── java│ │ │ └── org│ │ │ └── cerbos│ │ │ └── demo│ │ │ ├── config│ │ │ │ └── CerbosConfig.java│ │ │ ├── controller│ │ │ │ └── ProfileController.java│ │ │ ├── Main.java│ │ │ ├── model│ │ │ │ ├── Employee.java│ │ │ │ └── Profile.java│ │ │ └── repository│ │ │ ├── EmployeeRepository.java│ │ │ └── ProfileRepository.java│ │ └── resources│ │ └── application.yml Of course, your Java package names can be different. Ensure you follow the structure for cerbos-policies.

First, you can test the Cerbos policies locally. Open a Terminal at the root of the project and run the Cerbos container:

docker-compose up

You should see a result like this:

my-cerbos-container | Test results my-cerbos-container | └──profileTestSuite (profile_test.yml) [32 OK] my-cerbos-container | my-cerbos-container | 32 tests executed [32 OK] my-cerbos-container exited with code 0 As you can see, the tests are successful. Later, we will run the tests automatically in our CI pipeline. We needed this step to make sure that the policies were correct. Comment out this line ( command: compile /policies) to keep the container running.

Run docker-compose up again.

Check that Cerbos is up and running by opening http://localhost:3592/ in your browser.

You should see a page like this:

Start the Spring Boot application from your IDE.

Since we enabled the H2 Console, we can see that the database contains the expected dummy data:

Let’s try to access the profile with ID 111 as an employee with ID 123:

$ curl -X GET http://localhost:8080/api/profile/get/111/123 Employee(employeeId=123, name=John Doe, email=john.doe@me.com, salary=1500.0, role=employee) The profile belongs to this employee, so it works as expected.

Now let’s try the same, but this time using the ID of the HR:

$ curl -X GET http://localhost:8080/api/profile/get/111/456 Employee(employeeId=123, name=John Doe, email=john.doe@me.com, salary=1500.0, role=employee) As expected, it shows the details because the HR has access to all actions.

Now, let’s try the DELETE request:

$ curl -X DELETE http://localhost:8080/api/profile/delete/111/123 Forbidden As anticipated, the employee is not allowed to delete their profile.

Let’s try with the hr role:

$ curl -X DELETE http://localhost:8080/api/profile/delete/111/456 Profile deleted successfully So far, everything works fine.

However, what will happen if an employee attempts to access someone else’s profile?

Let’s try to access the profile of employee1 with the employee ID of employee2:

$ curl -X GET http://localhost:8080/api/profile/get/111/321 Employee(employeeId=123, name=John Doe, email=john.doe@me.com, salary=1500.0, role=employee) Since we haven’t defined any custom rules, employees can currently access each other’s profiles. This is not the desired behavior, as we want to keep the information secure.

Cerbos addresses this issue using Conditions, which utilize Common Expression Language (CEL) syntax. You can add attributes to the principal and the resources to evaluate and enforce specific access conditions. For instance, you can check if a user’s address is within a certain geographic location, etc.

request: principal: id: john roles: - employee attr: geography: GB Checking the condition:

condition: match: all: of: - expr: > "GB" in R.attr.geographies - expr: P.attr.geography == "GB" Let’s refine our Cerbos policies by incorporating conditions.

Modify the resources.yaml to add a new attribute called owner:

resources: profile#1: id: profile#1 kind: profile attr: owner: 123 Add this line to the profile.yaml for the read action:

- actions: - read effect: EFFECT_ALLOW roles: ["*"] condition: match: expr: (request.resource.attr.owner == request.principal.id) || ('hr' in request.principal.roles) The policy allows any role to access the resource, provided they meet the condition specified in the match expression. This condition allows access if either the principal is the owner of the resource or holds the hr role.

We also need to adjust the Java code. Extend the code that gets the Principal and Resource in the getProfile() method with this:

Principal principal = Principal.newInstance(employeeId, employee.getRole()).withAttribute("id", AttributeValue.stringValue(employeeId)); Resource resource = Resource.newInstance("profile", profileId).withAttribute("owner",AttributeValue.stringValue(String.valueOf(profile.getEmployee().getEmployeeId()))); * Principal: Represents the user requesting access. The “id” attribute helps to identify and match the principal in access control decisions. * Resource: Represents the entity being accessed. The “owner” attribute specifies the resource’s owner and facilitates access control checks.

This setup allows Cerbos to enforce policies based on these attributes. For example, it ensures that only the resource owner or users with specific roles (like ‘hr’) can access or modify the resource.

The PDP receives these values and determines if the action is allowed based on the predefined policy.

The initial test cases will no longer work after these changes. We can extend the test suites by introducing multiple user resources and principals. Replace the content of the profile_test.yml with this:

name: profileTestSuitedescription: Tests for verifying the profile resource policyprincipals: hr: id: hr1 roles: - hr employee1: id: emp1 roles: - employee employee2: id: emp2 roles: - employee employee3: id: emp3 roles: - employeeresources: profile: kind: profile id: emp1 attr: owner: emp1 profile2: kind: profile id: emp2 attr: owner: emp2tests: - name: profile actions input: principals: - employee1 - employee2 - employee3 - hr resources: - profile - profile2 actions: - create - read - update - delete expected: - resource: profile principal: employee1 actions: create: EFFECT_DENY read: EFFECT_ALLOW update: EFFECT_DENY delete: EFFECT_DENY - resource: profile2 principal: employee2 actions: create: EFFECT_DENY read: EFFECT_ALLOW update: EFFECT_DENY delete: EFFECT_DENY - resource: profile principal: employee3 actions: create: EFFECT_DENY read: EFFECT_DENY update: EFFECT_DENY delete: EFFECT_DENY - resource: profile principal: hr actions: create: EFFECT_ALLOW read: EFFECT_ALLOW update: EFFECT_ALLOW delete: EFFECT_ALLOW - resource: profile2 principal: hr actions: create: EFFECT_ALLOW read: EFFECT_ALLOW update: EFFECT_ALLOW delete: EFFECT_ALLOW Automating Policy Tests with Semaphore CIAutomating policy tests in the CI/CD pipeline is a best practice. This means any misconfigured rules will cause the tests to fail so that you can react quickly.

In this section, you’ll learn how to integrate and run Cerbos policy tests within the Semaphore CI pipeline.

Semaphore requires a sempahore.yml file. This configuration file specifies the tasks and workflows that Semaphore CI should execute. It outlines the steps for building your application, running tests, and deploying changes. Create a new directory in the root of your project called .semaphore. Create the semaphore.yml file inside it:

version: v1.0name: Cerbos Policy Executionagent: machine: type: e1-standard-2 os_image: ubuntu2004blocks: - name: Compile Policies task: jobs: - name: Compile commands: - checkout - docker run -it --name my-cerbos-container -v ./cerbos-policies:/policies -p 3592:3592 ghcr.io/cerbos/cerbos:0.34.0 compile /policies - docker logs my-cerbos-container The semaphore.yml file performs the following tasks:

• Sets up a virtual machine with ubuntu2004 image.
• Runs the Docker container to compile the policies using the Cerbos command compile /policies.
• Retrieves the container logs for review in the job’s output.

To run the tests in the pipeline, you need a free Semaphore account. Visit the signup page and choose either GitHub or Bitbucket for registration. In this tutorial, I will use GitHub.

Next, create a new project by clicking the “+ Create new” button.

I will connect my repository with Semaphore, which I used for this tutorial.

Semaphore will automatically initialize the project shortly. You can add more people to the project if you wish.

The final step is to create the workflow for the pipeline. With the semaphore.yml file already in place, you can use it directly for the workflow.

Let’s push something to the repository to trigger the pipeline.

For example, I added a README.md file to my project.

Shortly after the push, you’ll see that the pipeline will run the tests.

To see the details, click on the Compile job log.

As we added the docker logs command, we can access the container’s logs. There, you can see that all the tests were successful.

Let’s deliberately misconfigure the policies. For example, I’ll change the expected result for the hr role from EFFECT_ALLOW to EFFECT_DENY in one of the actions:

- resource: profile principal: hr actions: create: EFFECT_ALLOW read: EFFECT_DENY update: EFFECT_ALLOW delete: EFFECT_ALLOW Push the changes.

As expected, the job failed:

You can see in the logs which test failed:

ConclusionIn this tutorial, you learned what Cerbos is, and how to secure a Spring Boot application with Cerbos policies. You discovered how to create granular policies based on attributes to manage access control effectively.

Additionally, you set up automated policy testing with Semaphore CI. You can integrate this step into your existing pipeline.

By following this guide, you now have a solid foundation in leveraging Cerbos to enhance the security of your applications.

While this tutorial covered the basics of Cerbos, there are many more features to explore. For a deeper understanding, I encourage you to dive into the comprehensive Cerbos documentation.

You can find the source code of this tutorial in my GitHub repository.

Thank you for following along!

The post Securing a Spring Boot Application with Cerbos appeared first on Semaphore.

View Details

Marshalling client traffic to applications running in Kubernetes clusters, has for many years been the job of ingress controllers, using the Ingress API. However, the Ingress API has many limitations; it’s terse by design, ambiguous in places, and has no formal means for extending its limited capabilities. In short, without enhancement, it doesn’t meet the needs of the traffic management use cases in the real world. The Kubernetes community recognized the need for a better solution, and after several years in the making, this arrived in the form of the Gateway API.

The Kubernetes Gateway API reached GA status in October 2023 and continues to evolve as new features are identified and validated within the community. There are many improvements over the Ingress API:

• Role-based – multiple object types (GatewayClass, Gateway, HTTPRoute, TCPRoute, and so on) replace the monolithic Ingress object type, to better reflect the fact that multiple actors need to control the configuration of the end-to-end ingress experience
• Expansive – the Ingress API focuses on applications using the HTTP protocol, whereas the Gateway API caters for numerous protocols at different layers in the networking stack (HTTP/S, gRPC, TCP, UDP)
• Expressive – the limited traffic management features in the Ingress API have been augmented to include traffic splitting and mirroring, request and response header modification, HTTP redirects and rewrites, and much more
• Extendable – the unofficial, unregulated method of feature extension in the Ingress API, annotations, has been replaced with a more formal method of extension through ‘policy attachment‘

In this tutorial, you’ll use the Gateway API to configure ingress traffic to an example application running in a local Kubernetes cluster. You’ll be able to differentiate between the different Gateway API object types and see how they fit together to provide the full ingress experience for an application.

PrerequisitesThis tutorial uses a local development cluster using the Kind tool, where the cluster nodes are Docker containers. Docker is a prerequisite for using Kind. Follow the Quick Start guide to install Kind and to provision a local Kubernetes cluster.

To try and emulate operating in a cloud environment, the Cloud Provider for Kind should also be deployed. This will enable you to establish a local load balancer for use with the Gateway API.

Here’s a complete list of the prerequisites:

• Docker (macOS, Windows, Linux – Docker Desktop or Linux – Docker Engine)
• Kind
• Cloud Provider for Kind
• Kubernetes CLI – kubectl (macOS, Windows, Linux)

The Kubernetes configuration used in the tutorial can be found in the following GitHub repo:

• https://github.com/nbrownuk/gateway-api-tutorial

Deploy an ApplicationWith a Kubernetes cluster up and running, it’s time to deploy a sample application to the cluster, which we eventually hope to consume from a web browser, once we’ve configured ingress using the Gateway API. The app simply displays a static web page providing some information about its environment.

Step 1: Apply the Kubernetes Configuration for the AppThe application can be deployed to the cluster using kubectl:

kubectl apply -f \ https://raw.githubusercontent.com/nbrownuk/gateway-api-tutorial/main/nginxhello.yaml Step 2: Check the App’s StatusAnd, we can check that it’s started up correctly, by retrieving the Pod, Service and Deployment objects that have the label app.kubernetes.io/name=nginxhello. We should see something like this:

$ kubectl -n nginxhello get po,svc,deployNAME READY STATUS RESTARTS AGEpod/nginxhello-cbfb6bbb6-zcqqq 1/1 Running 0 9sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEservice/nginxhello ClusterIP 10.96.243.155 <none> 80/TCP 9sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/nginxhello 1/1 1 1 9s We’ve got a Deployment with a single replica Pod, which is fronted by a Service of type ClusterIP.

With the app up and running, it’s time to turn our attention to the configuration of the Gateway API.

Deploy a Gateway ControllerJust like the older Ingress API in Kubernetes, there is no in-tree controller to act on instance objects of the Gateway API that are applied to the cluster. Instead, we have to deploy a third-party gateway controller of our choice. If you’re using an Amazon EKS cluster, it’s likely to be the AWS Gateway API Controller. And, if you’re using Microsoft’s AKS, it’s likely to be the Application Gateway for Containers ALB Controller. But, there are a whole bunch of different cloud-native proxies that implement the Gateway API, too, including the Envoy Gateway. The Envoy Gateway is based on the Envoy proxy, and it’s what we’ll use in this tutorial as the implementation of the Gateway API.

Step 1: Install the Envoy GatewayUnlike the Ingress API, which is a constituent part of the default set of Kubernetes APIs exposed by the API server, the Gateway API is provided as Custom Resource Definitions (CRDs). That means the CRDs need to be installed before they can be used. Sometimes, this installation is performed as part of the installation of the chosen gateway controller. Installing the Envoy Gateway gives us the CRDs, as well as the controller itself. There are a few ways to install the Envoy Gateway, but the project provides a handy install YAML file as part of its GitHub releases:

kubectl apply --server-side -f \ https://github.com/envoyproxy/gateway/releases/latest/download/install.yaml It will take a few moments before the Envoy Gateway is ready in the cluster.

Step 2: Check the Status of the Envoy GatewayThe installation should give us the Envoy Gateway deployed to a new namespace in our cluster, envoy-gateway-system, which we can check by retrieving the Pod, Service and Deployment objects in that namespace:

$ kubectl -n envoy-gateway-system get po,svc,deployNAME READY STATUS RESTARTS AGEpod/envoy-gateway-654d5d67f8-pzmr6 1/1 Running 0 34sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEservice/envoy-gateway ClusterIP 10.96.79.24 <none> 18000/TCP,18001/TCP,18002/TCP,19001/TCP 34sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/envoy-gateway 1/1 1 1 34s If the Pod status is Running, all should be well with the gateway controller.

Step 3: Create a GatewayClassBut, it’s possible to run more than one gateway controller in a cluster; different controllers for different purposes, or horses for courses. So, we must associate the gateway controller with a GatewayClass object that can be referenced by other Gateway API objects we want to create, so that the correct controller is selected for the job. The GatewayClass definition that we need for the Envoy Gateway looks like this:

```

gateway-class.yaml---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: envoy-gatewayspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller

`` The value of thecontrollerNamefield is the same as the default name associated with the Envoy Gateway controller. We can see this by inspecting theConfigMap` that is used to configure the Envoy Gateway during installation:

$ kubectl -n envoy-gateway-system get cm envoy-gateway-config \ -o "jsonpath={.data['envoy-gateway\.yaml']}"apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyGatewaygateway: controllerName: gateway.envoyproxy.io/gatewayclass-controllerlogging: level: default: infoprovider: type: Kubernetes Note the value of the controllerName in the EnvoyGateway definition. So, the GatewayClass object definition we’re about to use has a correlation with the Envoy Gateway by virtue of the controllerName. But, it needs to be applied to the cluster to take effect:

kubectl apply -f \ https://raw.githubusercontent.com/nbrownuk/gateway-api-tutorial/main/gateway-class.yaml Step 4: Examine the GatewayClass ObjectOnce the cluster-scoped object has been created in the cluster, we can check its status:

$ kubectl get gc envoy-gatewayNAME CONTROLLER ACCEPTED AGEenvoy-gateway gateway.envoyproxy.io/gatewayclass-controller True 7s It should have an ‘Accepted’ status of ‘True’ to indicate the Envoy Gateway has accepted processing on behalf of the GatewayClass. We’re now set up to make use of the Envoy Gateway to handle ingress traffic to the cluster.

Configure a GatewayThe next part of the process is to configure another Gateway API component for our scenario; a Gateway object.

Step 1: Define a GatewayA Gateway defines a set of ‘logical endpoints’ that are associated with an IP address. You can think of a Gateway as defining the characteristics of the traffic that can enter the cluster at the edge. Gateway controllers act on Gateway definitions to allow the ingress of traffic to the cluster. Depending on its type, the gateway controller might provision cloud infrastructure resources, or configure a proxy on behalf of the Gateway. The endpoints can be addressed by clients internal or external to the cluster, that wish to consume the apps running in the cluster. The logical endpoints are called ‘Listeners’, and are constituent parts of Gatewaydefinitions:

```

gateway.yaml---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: http-gw namespace: nginxhellospec: gatewayClassName: envoy-gateway listeners: - name: http protocol: HTTP port: 80

`` Listeners can define different network protocols; HTTP, HTTPS, TCP and UDP. Here, we’ve got a Listener specifying the HTTP protocol for traffic destined for theGatewayon port 80. Notice also that theGatewayreferences the Envoy Gateway instance with the associatedGatewayClassnamedenvoy-gateway`. It’s this gateway controller that acts on its behalf.

Step 2: Create a GatewayThe Gateway definition needs to be applied to the cluster:

kubectl apply -f \ https://raw.githubusercontent.com/nbrownuk/gateway-api-tutorial/main/gateway.yaml Once the API server has created the object, the Envoy Gateway acts on the definition to enable the ingress of HTTP traffic on port 80. But, what actions does the Envoy Gateway take? We can check in on the envoy-gateway-system namespace to see:

$ kubectl -n envoy-gateway-system get po,svc,deploy \ -l gateway.envoyproxy.io/owning-gateway-name=http-gwNAME READY STATUS RESTARTS AGEpod/envoy-nginxhello-http-gw-ca0a3c04-55ddfb67fc-6njpv 2/2 Running 0 16sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEservice/envoy-nginxhello-http-gw-ca0a3c04 LoadBalancer 10.96.21.92 172.18.0.3 80:32597/TCP 16sNAME READY UP-TO-DATE AVAILABLE AGEdeployment.apps/envoy-nginxhello-http-gw-ca0a3c04 1/1 1 1 16s There are some additions to the namespace that have been brought about by the Envoy Gateway. Now there is a replicated deployment of the Envoy proxy, exposed via a LoadBalancer type Service, with an externally addressable IP address on port 80. The IP address, provisioned by the Cloud Provider for Kind, is 172.18.0.3. Be sure to note which EXTERNAL-IP your env has given you – you’ll need it in the next steps! Clients can send HTTP requests to this IP address when they need to communicate with an app running in the cluster. The set up looks something like this:

Step 3: Examine the Gateway ObjectWe can also check the status of the created Gateway object:

$ kubectl -n nginxhello get gtw http-gwNAME CLASS ADDRESS PROGRAMMED AGEhttp-gw envoy-gateway 172.18.0.3 True 42s This tells us the IP address associated with the Gateway (it’s the same as the externally addressable LoadBalancer Service), and that it has been programmed into the data plane by the Envoy Gateway.

But, which app do client requests get sent to? We haven’t yet defined a route to a backend service running in the cluster.

Define RoutesThe final piece of the jigsaw is to define an HTTPRoute object that will allow traffic to be sent to the application that we deployed right at the beginning of this tutorial.

Step 1: Define an HTTPRouteThe HTTPRoute will look like the following:

```

httproute.yaml---apiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata: name: nginxhello namespace: nginxhellospec: parentRefs: - name: http-gw hostnames: - 172-18-0-3.nip.io # to be replaced with your EXTERNAL-IP rules: - backendRefs: - name: nginxhello port: 80

`` Firstly, there is a reference to theGatewaythat will be responsible for this route, the one we’ve just established in the cluster, calledhttp-gw. TheHTTPRoutewill be ‘attached’ to the HTTP Listener in theGateway. Then, we define a list of hostnames that must match an incoming HTTP request host header. There is a single entry;172-18-0-3.nip.io1. We’re using the nip.io service to give us a domain name for the IP address of the LoadBalancer service associated with the Gateway in our local environment. Finally, we have a single rule, which applies no conditions, and simply references theClusterIP` Service that fronts our application. Any client HTTP requests addressed to the domain name will get routed to the application as a result.

Step 2: Create an HTTPRouteThe HTTPRoute needs to be applied to the cluster.

kubectl apply -f \ https://raw.githubusercontent.com/nbrownuk/gateway-api-tutorial/main/httproute.yaml Step 3: Examine the HTTPRoute ObjectThe first check we should make is to ensure the HTTPRoute object was created successfully:

$ kubectl -n nginxhello get httproute nginxhelloNAME HOSTNAMES AGEnginxhello ["172-18-0-3.nip.io"] 11s This response doesn’t tell us too much, other than the object has been created successfully. What is of more importance to us, is whether the HTTPRoute has been accepted by the Gateway, and we need to get more information by describing the object:

kubectl -n nginxhello describe httproute nginxhello We’ll get a lot of information back, but if we look out for the ‘status’ information provided, we should see something like this:

<snip>Status: Parents: Conditions: Last Transition Time: 2024-05-29T13:41:39Z Message: Route is accepted Observed Generation: 1 Reason: Accepted Status: True Type: Accepted<snip> The status tells us that the HTTPRoute has been accepted by the Gateway, and is now attached to it for processing purposes. Our scenario is now complete:

We’re good to go!

Step 4: Consume the ApplicationThe final step is to use a web browser to attempt to consume the application, by using the hostname specified in the HTTPRouteas the URL. It’s a non-TLS connection, so do accept the security exception, and the app should respond to the request accordingly:

ConclusionThis tutorial provides a basic introduction to the capabilities of the Gateway API. It has much more to offer and represents a big improvement on the frailties of the Ingress API for ingress scenarios. It still has a way to go before it reaches full maturity, but it’s already mature enough to be used in production clusters. If you’re considering moving your existing Ingress definitions over to the Gateway API, check out the Ingress2Gateway utility, which provides translation for a growing set of ingress providers.

If the IP address of your Gateway is different, be sure to amend the hostname accordingly. ↩

Learn more:

• Blue-Green Deployments With Kubernetes
• Top 10 Must-Have Tools for Kubernetes Engineers
• Guide to the Rolling Update Deployment Strategy

The post How to Expose Kubernetes Apps Using the Gateway API appeared first on Semaphore.

View Details

Search functionality is key to the success and usability of any modern application, as users want to find relevant information quickly and efficiently. Over time, Elasticsearch has become the go-to search solution for many organizations. However, despite its popularity and broad feature set, it may not be the best option for startups due to its complexity, cost, and steep learning curve. No wonder startups are interested in more agile Elasticsearch alternatives.

In this article, you will explore why startups should look into Elasticsearch alternatives for their search solution. Then, you will see the key aspects to consider when comparing search solutions and apply them to the top 7 alternative search services for startups.

Let’s dive in!

Why Elasticsearch May Not Be the Best Search Solution for a StartupElasticsearch is the most popular distributed, RESTful search and analytics engine. It falls into the category of search solutions, which provide powerful full-text and vector search capabilities over large datasets, along with features for logs, metrics, and APM (Application Performance Monitoring).

The end goal is to enable users to quickly find relevant information, even with typos in their search queries. Admins can then analyze search data to find patterns, make decisions, and improve business processes.

Integrating an Elasticsearch-powered search feature in your application results in improved user experience and efficiency. Despite these benefits, Elasticsearch may not be the best choice for startups for three key reasons:

• High costs: Handling an Elasticsearch architecture on platforms like AWS can become increasingly costly as data volumes grow.
• Too many features: While beneficial for large enterprises, the extensive feature set of Elasticsearch can be overwhelming and unnecessary for smaller startups.
• Difficult setup and steep learning curve: Setting up Elasticsearch and getting the best out of it requires significant expertise and time.

Startups generally need affordable, simple, and flexible solutions that are easier to set up and maintain. The technology stack should evolve quickly alongside the core business, without the burden of complex infrastructure management. This is why they love to explore some Elasticsearch alternatives.

Aspects to Consider When Evaluating Search ServicesNow, assume you are a startup looking for a good Elasticsearch alternative. These are the main elements to keep in mind when comparing search solutions:

• Type: Is it open-source, proprietary, self-hosted, or a cloud service?
• Search types: What are the underlying algorithms? (e.g., vector search, geo search, nearest neighbor search, etc.)
• Features: The list of the main features provided by the search service.
• Pros and cons: The advantages and drawbacks of the solution.
• API clients: The available API client libraries (and programming languages) to interact with the search engine and simplify data retrieval.
• Popularity: Useful statistics about the solution, such as the review score, number of GitHub stars, downloads, installations, users, and more.
• Pricing: The pricing model of the solution, in case of a premium service.

Top 7 Elasticsearch Alternatives for StartupsLet’s use the criteria presented above to list and compare the best Elasticsearch alternatives for startups.

If you are eager to find out what these solutions are and what they offer, take a look at the table below:

| Feature | Algolia | Typesense | ZincSearch | OpenObserve | OpenSearch | Vespa | | --- | --- | --- | --- | --- | --- | --- | | GitHub stars | — | 18.6k | 16.7k | 10.2k | 9k | 5.4k | | First public commit | 2012 | 2015 | 2021 | 2023 | 2014 | 2016 | | Programming language | C++ | C++ | Go | Rust | Java | Java, C++ | | Source code | Proprietary | Open source | Open source | Open source | Open source | Open source | | Deployment | Cloud | Self-hosted/Cloud | Self-hosted | Self-hosted/Cloud | Self-hosted/Cloud | Self-hosted/Cloud | | Pricing | Premium | Free/Premium | Free | Free/Premium | Free | Free/Premium | | Search types | Hybrid, semantic, and more | Vector, semantic, and more | Full-text | Log | Fuzzy, wildcard, and more | Fuzzy, vector, and more | | API clients | PHP, JS, Ruby, Python, iOS, Android, C#, Java, Go, Scala, Rails, Symfony, Django, Laravel, Kotlin, Elixir, ColdFusion, Erlang, .NET | JavaScript, PHP, Python, Ruby, Go, .NET, Java, Rust, Dart, Perl, Swift, Clojure | Go, Python, JavaScript | — | Python, Java, JavaScript, Go, Ruby, PHP, .NET, Rust | Python, Java |

Time to explore the seven best startup-oriented search solutions!

Algolia is a cloud-based, hosted search engine that provides real-time search results through a comprehensive API interface. It consists of two main components:

1. Search Implementation: Enables developers to create and maintain excellent search experiences for users.
2. Search Analytics: Empowers business teams to analyze and optimize these experiences to meet evolving business objectives.

Algolia is an end-to-end AI search and discovery platform that offers fast and reliable search capabilities in more than 68 languages. Its cloud-native nature makes it more agile and easier to roll out than Elasticsearch. This is especially important for startups, which are always looking for quick setups and integrations.

Explore the official site to find out why Algolia is a good Elasticsearch alternative.

📚 Type: End-to-end AI search and discovery platform

🔎 Search types: Hybrid keyword and vector search, semantic search, faceted search, and geo search via NeuralSearch

🛠️ Features:

• Support for over 68 languages
• Indexing, auto complete, and stemming
• Configurable typo tolerance
• Highlighted results to show which section and words matched user queries
• Advanced language processing, like options for plurals handling, stop words, optional words, and prefixing behavior
• Synonym handling
• Support for location-based queries with geo-aware search results
• Multiple sorting strategies
• Grouping and deduplication of records
• Advanced personalization capabilities so that different users get different results for the same queries
• Search analytics available in a visual dashboard
• AI recommendations

👍 Pros:

• Average response time of 1 to 20 milliseconds for query processing
• Over 100 releases per year
• 99.999% uptime guarantee via SLA
• Dozens of community integrations with CMS and e-commerce platforms

👎 Cons:

• Proprietary search engine with vendor lock-in
• Can become expensive, especially on projects with high search volumes and large datasets

🔗 API clients:

• Official API clients for PHP, JavaScript, Ruby, Python, iOS, Android, C#, Java, Go, Scala, Rails, Symfony, Django, Laravel, Kotlin
• Community API clients for Elixir, ColdFusion, Erlang, .NET

📊 Popularity:

• 17k paying customers
• 1.7+ trillion searches every year
• 500k+ developers using Algolia
• 30 billion indexed records
• 4.4/5 on Gartner Peer Review
• 4.5/5 on G2
• 4.7/5 on Capterra

💰 Pricing:

• Build:
  • Free plan
  • Up to 10k search requests/month, with 1m records included
• Grow:
  • Free/pay-as-you-go plan
  • Free up to 10k search requests/month, then $0.50 per additional 1k search requests
  • Free up to 100k records, then $0.40 per additional 1k records/month
• Premium:
  • Search experience with some AI features
  • Contact sales for pricing
• Elevate:
  • Full AI search offering
  • Contact sales for pricing

TypesenseTypesense is a typo-tolerant search engine optimized for instant search-as-you-type experiences. It serves as an open source alternative to Algolia—addressing some of its key issues—and represents a batteries-included alternative to ElasticSearch.

The team designed every feature from the ground up, with a close eye on making it intuitive and easy to use. This makes Typesense ideal for startups seeking an easy-to-integrate, quick-to-setup, and fast search solution.

Discover how Typesense compares to alternatives.

📚 Type: Open source, typo-tolerant search engine

🔎 Search types: Vector search, semantic search, geo search, fuzzy search, conversational search, image and voice search

🛠️ Features:

• Typo tolerance
• Meticulously architected from the ground up for low-latency instant searches.
• Support for dynamic sort results based on a particular field at query time
• Advanced faceting and filtering capabilities
• Support for grouping results and getting distinct records
• Support for federated search across multiple collections in a single HTTP request
• Ability to search and sort by results around a latitude/longitude or within a bounding box
• Ability to send questions and receive responses with fully-formed sentences, as with ChatGPT, but over your own data.
• Possibility to search through images using text descriptions
• Capture and send queries via voice recordings
• Connect one or more collections via common reference fields and join them during query time.
• Synonym handling
• Curation and merchandizing to boost particular records to a fixed position in the search results

👍 Pros:

• Batteries-included developer experience with a 30-second setup
• Several examples with code covering different real-world use cases
• Integration with Firebase, AWS DynamoDB, MongoDB, Supabase, Laravel, WordPress, Magento, and popular documentation technologies
• In-depth guides to simplify migration from Algolia

👎 Cons:

• No out-of-the-box user-level search personalization
• Implementing search recommendations requires custom implementation

🔗 API clients:

• Official API clients in JavaScript, PHP, Python, Ruby
• Community API clients in Go, .NET, Java, Rust, Dart, Perl, Swift, Clojure

📊 Popularity:

• 18.6k stars on GitHub (41 committers)
• 14 million docker pulls
• 5 billion cloud searches per month
• 5/5 on G2

💰 Pricing:

• Open Source (self-hosted): Free via a GPL license
• Typesense Cloud: Prices change based on memory and CPU resources

ZincSearchZincSearch is an open source and lightweight alternative to Elasticsearch. It provides a search engine written in Go that requires low resources and is designed for full-text indexing. Under the hood, it relies on bluge as the underlying indexing library.

ZincSearch represents a free drop-in replacement for Elasticsearch if you are searching using Kibana and ingesting data using Ingest APIs. Note that Kibana is not supported with ZincSearch, as this provides its own basic UI for data exploration.

📚 Type: Open source search engine for full-text indexing

🔎 Search types: App search

🛠️ Features:

• Full-text indexing capabilities
• Web UI for querying data written in Vue.js
• Compatible with Elasticsearch APIs for data ingestion
• Out of the box authentication system
• Schemaless data representation
• Index storage in disk
• Support for data aggregation

👍 Pros:

• Easy installation through binary files for multiple platforms
• Docker image for quick setup
• Complete configuration in about 2 minutes
• Simple, developer-oriented, lightweight tool

👎 Cons:

• Not designed for log search
• No high availability architecture

🔗 API clients:

• Official API clients in Go, Python, JavaScript

📊 Popularity:

• 16.7k stars on GitHub (57 committers)

💰 Pricing: Free

OpenObserveOpenObserve is an open source, cloud-native log search, infrastructure monitoring, and APM solution. It is designed to handle logs, metrics, traces, and analytics, offering full compatibility with the OpenTelemetry standard.

You can think of OpenObserve as a mix of Prometheus for metrics, Elasticsearch for logs, Jaeger for traces, and Grafana for dashboards. Thanks to its compatibility with several storage services, it can significantly reduce storage costs—promising to lower them to about 140 times less than Elasticsearch.

Check out the docs to see why OpenObserve is a good alternative to Elasticsearch.

📚 Type: Open source and cloud-native observability platform built specifically for logs, metrics, traces, and analytics

🔎 Search types: Log search

🛠️ Features:

• Support for storing and exploitation of logs, metrics, and traces
• Full compatibility with the OpenTelemetry standard
• Real User Monitoring (RUM) via performance tracking, error logging, and session replay
• Over 14 different chart types for comprehensive data visualization
• Advanced ingest and query functions for data redaction, log reduction, and compliance
• Support for querying logs and traces via SQL and metrics via SQL and PromQL
• Advanced embedded GUI with an intuitive interface
• Versatile storage options, including local disk, S3, MinIO, GCS, Azure Blob Storage
• High availability and clustering to ensure reliable and scalable performance
• Dynamic schema to adapt your data structure seamlessly
• Built-in authentication with RBAC privilege system
• UI available in 11 languages, including English, Spanish, German, French, Chinese, and more

👍 Pros:

• Open source
• Docker image available for easy setup
• First configuration in less than 2 minutes
• Integration with Slack, Amazon S3, Microsoft Teams, Google Cloud Storage, and Minio

👎 Cons:

• Not so easy to integrate with other technologies
• Limited to log search

🔗 API clients: None

📊 Popularity:

• 10.2k stars on GitHub (53 committers)

💰 Pricing:

• Open Source (self-hosted): Free via an AGPL license
• Open Source (self-hosted): Paid via an enterprise license
• OpenObserve Cloud: 4 different plans, from a free plan for developers to a plan dedicated to enterprises

OpenSearch

OpenSearch is an open source software suite for search, analysis, and observability. Forked from Elasticsearch 7.10, it maintains backward compatibility with Elasticsearch’s REST API, query syntax, and responses. That makes it the most straightforward drop-in solution among the Elasticsearch alternatives in this list.

OpenSearch allows users to seamlessly ingest, protect, search, aggregate, visualize, and analyze data, just as in Elasticsearch, but with all the benefits of open source. Its community nature and origins make it ideal for startups, as you will find many users to learn and share lessons learned with.

Read more about why OpenSearch is not Elasticsearch.

📚 Type: Open source search and analytics suite

🔎 Search types: Fuzzy search, wildcard search, geo search, vector search, neural search

🛠️ Features:

• Application and infrastructure monitoring with log data storage and automated alerts
• Centralized logs for real-time security monitoring and forensic analysis
• Observability capabilities through logs, metrics, and traces for real-time application and business monitoring
• Anomaly detection to identify and notify about atypical data
• KNN (K-nearest neighbors) for finding nearest neighbors in vector data
• Performance analyzer to monitor and optimize cluster performance
• SQL and piped processing language support for querying data
• Advanced indexing capabilities
• Asynchronous search for running background search requests
• Cross-cluster replication for high availability and disaster recovery
• Extensible architecture with dozens of plugins available

👍 Pros:

• Community-oriented project with a lot of active users on GitHub and Slack
• Easy setup via Docker
• Highly fast and scalable
• OpenSearchCon to meet the community and see the search suite in action

👎 Cons:

• Can be complex to be configured for optimal performance
• Same learning curve as Elasticsearch

🔗 API clients:

• Official API clients in Python, Java, JavaScript, Go, Ruby, PHP, .NET, Rust, Rust, Hadoop

📊 Popularity:

• 9k stars on GitHub (302 committers)
• 4.2/5 on G2

💰 Pricing: Free

Vespa

Vespa is an open source search engine and vector database. Originally launched as a pioneering search engine in 1997, it became part of Yahoo through the Overture deal in 2003 and was open sourced in 2017.

Vespa supports vector search, lexical search, and search in structured data, all within the same query. Its integrated machine-learned model inference enables real-time application of AI, making it great for startups needing to quickly derive insights from data.

Delve into the relationship between Vespa, Elasticsearch, and Solr.

📚 Type: Open source search engine and vector database

🔎 Search types: Fuzzy search, vector search, geo search, lexical search, streaming search

🛠️ Features:

• Ability to add, modify, and remove documents with durable write acknowledgments in milliseconds
• Automatic data corruption repair from uncorrupted replicas
• Simple HTTP/2 API for writing and retrieving data
• Response times typically in tens of milliseconds
• Rich data schemas that allow various primitive types, collections, structs, and tensors
• Referencing and querying fields from referenced documents can be done without performance penalties
• Queries support with structured filters, free text, and vector search operators
• Streaming search mode for low-latency access to billions of documents per node
• Result ranking via configurable functions, including machine-learned models and ONNX integration
• Indexing with linguistic transformation
• Support for multiple ranking phases and advanced text ranking features
• Support for nested groupings and data aggregation
• Support for multiple clusters of any size with fault tolerance and automatic node routing
• Tokenization and stemming in English, German, French, Spanish, and Italian

👍 Pros:

• Easy installation via RPM files, Docker image, or deployment in AWS
• Application configuration specified in a single application package for simplified deployment
• Multi-tenant architecture with no single points of failure
• Real-time data ingestion, processing, and streaming

👎 Cons:

• Confusing documentation
• Only a few languages fully supported, limiting the company’s growth in international markets

🔗 API clients:

• Official API clients in Python and Java

📊 Popularity:

• 5.4k stars on GitHub (79 committers)

💰 Pricing:

• Open Source (self-hosted): Free via an Apache 2.0 license
• Vespa Cloud: 3 different plans, from a basic plan for individuals to a plan dedicated to enterprises

Solr

Solr is an enterprise search platform built on Apache Lucene, an open source, Java-based, information retrieval library. It supports advanced full-text search with advanced filtering capabilities. Solr is suitable for startups that need to handle scalable volumes of traffic.

It is based on open standards and is designed to drive powerful document retrieval applications. In particular, it responds with structured documents to simple HTTP requests.

📚 Type: Open source enterprise search platform built on Apache Lucene

🔎 Search types: Wildcard search, spatial search, dense vector search, fuzzy search, proximity search, range search

🛠️ Features:

• Documents can be added via JSON, XML, CSV, or binary over HTTP
• Querying via HTTP with results in JSON, XML, CSV, or binary
• Advanced full-text search capabilities
• Comprehensive built-in admin UI
• Support for near real-time indexing for quick updates
• Extensible architecture that supports plugins for custom functionalities
• High schema flexibility with both schemaless and configured schema modes
• Powerful faceted search and filtering capabilities.
• Built-in support for geospatial search for location-based queries
• Advanced configurable text analysis in several languages

👍 Pros:

• Standalone search server with a REST-like API interface
• Based on open interfaces like XML, JSON, and HTTP
• Ideal for handling high-volume traffic at large scales
• High performance and scalability

👎 Cons:

• Hard-to-explore documentation that can be overwhelming for new users
• Confusing query language

🔗 API clients:

• Official API client library in Java
• 3rd-party API clients in Python, Ruby, PHP, C#/.NET, Scala, Perl, JavaScript, Clojure, Go, Rust, R, C++

📊 Popularity:

• 1.1k stars on GitHub (387 committers)
• 4.1/5 on G2
• 4.3/5 on Capterra

💰 Pricing: Free

ConclusionIn this guide, you learned why Elasticsearch may not be the best search solution for a startup. Fortunately, there are more affordable, open source, and easier-to-configure technologies for searching data. Here, you explored the seven best Elasticsearch alternatives with quick setups and low resource usage. As a startup, knowing these affordable solutions can help you save a lot of time and money!

The post 7 Elasticsearch Alternative Search Solutions for Your Startup appeared first on Semaphore.

View Details

Flaky tests—those intermittent failures that undermine confidence in test suites—are a persistent thorn in the side of development teams. As software complexity grows, so does the challenge of ensuring test reliability. In this episode, Srivishnu Ayyagari, a senior product manager at LambdaTest, offers valuable insights into the root causes of flaky tests and strategies to overcome them.

Edited transcriptionThe type of application and use case defines the testing strategy, says Srivishnu Ayyagari, senior product manager at cloud-based testing platform LambdaTest. For example, in e-commerce applications, “elements or components stay in the same place”, and consequently, require developers to “run the same test case every day.” In these cases, automated testing has become vital, allowing testers to automate the execution of tests on “different browsers and validate their use cases” through test scripts.

Aware of the potential of test automation, the tooling market continues developing software that further increases the reach of what’s possible to automate. In this way, since Srivishnu arrived at LambdaTest, he has been part of their transition from developing manual to automated testing tools. As he explains, working with “different kinds of open source frameworks like Selenium, Cypress, and Playwright” has allowed LambdaTest to “create a plethora” of valuable tools for QA professionals, including “Hyper Execute, Insight and SmartUI for visual regression testing and real-device mobile app testing.”

Understanding —and addressing— test instabilityFlaky tests (tests that intermittently pass or fail without any code changes) are the Achilles heel of test automation. The time burden of determining if a test failure indicates a bug or a false positive collides with the swiftness of automation. Furthermore, flaky tests seem unavoidable in today’s development, due to their tight correlation with current practices and architectural choices.

The rise of flaky tests, Srivishnu explains, goes hand in hand with the popularity of microservices architecture, in which “there’ll be different kinds of service, but there’ll be one single application, client application, which will be talking to different kinds of APIs.” Each of these APIs “might not perform at the same level compared to others;” moreover, different services might have varying reliance on external APIs or systems, which can introduce unpredictability. Besides, asynchronous operations or race conditions can lead to inconsistent results, as well as differences in testing environments (local, CI/CD, cloud).

Another root cause behind flakiness is a combination of asynchronous loading time of page elements and test script execution speed. The time it takes for elements to appear on a webpage can vary due to network conditions, device performance, and other factors —for example, the use of third-party features that have to be retrieved from external websites. Test scripts, in turn, often execute faster than page loading, leading to issues when trying to interact with elements that aren’t fully loaded.

In this case, mitigation strategies involve adding deliberate delays in test scripts to ensure elements are loaded before interacting with them and configuring the browser to wait for a specific amount of time before timing out. Likewise, it also helps to improve website performance and reduce the time it takes for elements to load.

Network latency, or the time it takes for data to travel between different locations, is another significant contributor to flaky tests. When applications and their components are distributed across multiple data centers, network performance can dramatically impact test reliability. “For example, a test might pass when running from a location close to the application’s server but fail when executed from a remote location due to increased network latency,” Srivishnu explains. To mitigate network latency, he recommends isolating tests and ensuring adequate network bandwidth. Besides, retrying failed tests will help differentiate between genuine application issues and network-related flakiness.

In all situations, when a “flaky test is found,” says Srivishnu, “you have to take a retry” to “check whether it is a correct result or is in the false positive result.” Here, repetitive failing after consecutive runs means “it is a hundred percent feature problem” that the developers need to address and fix. In turn, if it fails randomly after several runs, “that means that there’s an issue with the execution of the browser test,” and the solution involves checking “whether that command is being called correctly every time” and adding mitigation steps. In this regard, many testing frameworks now include built-in retries to handle transient failures.

Addressing flaky tests with LambdaTestSrivishnu encourages proactively addressing flaky tests in a test suite such as LambdaTest. Early detection and mitigation are key to maintaining healthy development cycles. “Once your automation tests start growing, it will be very hard to manage and see how many of the tests are giving false positive results,” he points out.

Beyond simply identifying flaky tests, understanding the underlying causes is part of remediation. To this end, LambdaTest features error categorization insights. “Providing users with just information about flaky tests wasn’t enough,” explains Srivishnu. “We wanted to give them deeper insights into the specific errors causing the flakiness.” Lambda Test’s error categorization feature breaks down flaky tests by the type of error encountered, such as “no such element” or “stale element reference” so developers can learn which are the most prevalent issues and prioritize their efforts.

Additionally, LambdaTest uses AI to identify patterns in test failures and flag potential flaky tests. To find these patterns, the platform tries combinations of different variables, such as system OS, browsers, and resolutions. At this point, LambdaTest provides data on the frequency and severity of flaky tests on these configurations and if they exceed the flaky threshold. LambdaTest also uses HyperExecute to isolate tests and limit the interference of environmental factors and test dependencies.

The effectiveness of flaky test detection can vary depending on the testing framework used. “The approach to flaky test detection is multifaceted,” says Srivishnu. “Different frameworks have their strengths, and it’s essential to choose the right tools and strategies based on project requirements and team preferences.” As such, while there are common strategies, specific implementations differ across tools. Playwright, for example, employs a retry mechanism to handle potential flakiness. If a test fails, it’s automatically retried a specified number of times. If the test passes on subsequent attempts, it’s flagged as potentially flaky. For Selenium-based testing, tools like Report Portal can be integrated to analyze test logs and identify patterns indicating flakiness. Cypress, another popular framework, offers its cloud-based solution for detecting flaky tests and providing insights into the specific commands causing issues.

Emerging testing frameworks and trendsBeyond automated testing, the broader context of UI technologies has implications for QA strategies. While React remains dominant, other frameworks have gained traction, each offering unique advantages. “We have seen adoption starting recently with the launch of Next.js, where most of the front-facing pages, so let’s say customer-facing pages, are mostly driven with Next.js.” Vue.js, for its part, provides a distinct architectural approach that resonates with many developers.

Concurrently, the testing framework ecosystem is also transforming. Playwright has emerged as a strong contender, distinguishing itself with its WebSocket-based communication, which significantly reduces latency compared to traditional HTTP-based approaches. As Srivishnu explains, “Playwright has seen a huge massive adoption being increasing and competing directly with Cypress as well as Selenium, where the major, game-changing thing that they have launched was the WebSocket connection.” Selenium, a long-standing industry stalwart, is responding to these developments by exploring a more interactive, by-directive approach to keep pace with modern testing requirements.

A key question regarding Playwright’s WebSocket-based approach is browser compatibility. Srivishnu clarifies that WebSocket support is indeed provided by the browsers themselves. “These are exposed by the browsers themselves to have a WebSocket-based interaction,” he explains.

Srivishnu strongly advocates for Playwright as the best starting point for teams looking to automate their test suites across various frameworks. “Playwright is JavaScript natively and JavaScript is very easy to code and get to know like prod as well,” he emphasizes.

Beyond its accessibility, Playwright offers a rich set of features for browser manipulation and testing. Its advanced command set and active open-source community further solidify its position as a preferred choice. “The best thing with Playwright is that all the commands or let’s say the APIs are much more advanced and they can do a lot of browser manipulation, DOM manipulation, like capture a screenshot, do a visualization test…” Srivishnu explains.

Another key advantage of Playwright is its speed. Through WebSocket connections, Playwright can significantly reduce test execution time compared to Selenium. “Playwright runs faster than Selenium because of that specific WebSocket feature,” Srivishnu affirms. Faster test execution translates to quicker feedback loops and increased developer productivity.

The bottom lineVisit Lambdatest.com to learn more about the platform and its resources, including a learning hub for testing, webinars, and test automation certifications. Connect with Srivishnu on Linkedin.

The post Srivishnu Ayyagari On Flaky Tests Anti-Patterns appeared first on Semaphore.

View Details

Traditional database hosting services, like cloud services, offer excellent performance but come with significant costs. However, not all projects require many dedicated resources. Considering that developers are constantly seeking affordable solutions, the database hosting landscape has evolved significantly in recent years. Thanks to the emergence of new players, numerous low-cost alternatives are now available for individuals and businesses considering database migration.

In this article, you will dig into popular database services to understand what they offer and how much they cost. Then, you will explore five cost-effective alternatives that guarantee affordability without compromising on functionality.

Let’s dive in!

The Real Cost of Database HostingIn most cases, databases themselves do not cost money since they are open-source technologies. This means you do not have to pay for the database software itself. Instead, the primary cost comes from hosting.

Database hosting services provide essential infrastructure, security, and maintenance to keep your database operational. These usually charge based on how large the servers used to host your databases are.

Additionally, there may be fees for database migration, when transferring data from one server to another. That is common with cloud providers. These fees, also known as egress fees, are typical in cloud storage systems like S3 but might also apply to databases. For more information, read our guide on how to avoid egress fees with Cloudflare R2.

Understanding the real costs behind databases is crucial for budgeting effectively and selecting cost-efficient solutions that fit your needs.

Database Hosting Services: Types, Cost, and ProvidersDatabase hosting is the process of storing databases on remote servers provided by a third-party company. That enables individuals and companies to access and manage their databases without needing to own and operate the physical hardware themselves.

Typically, database hosting is a service offered by a specialized provider. This service provides everything required to set up and run your chosen database. It offers a secure and accessible environment to keep your data safe and ensures scalability, allowing your data operations to grow as needed.

These are the more popular types of database hosting services:

• Integrated into web hosting
• VPS
• Cloud
• Serverless

Time to explore them by explaining what they are, analyzing how much they cost, and mentioning some key providers.

Built-In Databases in Traditional Web HostingMost traditional web hosting providers offer users the ability to create and manage pre-configured databases like MySQL or PostgreSQL. These databases are directly integrated into the hosting environment. Users can quickly set up them using tools like cPanel. Similarly, they can manage their data with clients like phpMyAdmin.

Save money with our guide on low-cost web hosting alternatives.

• Cost range: A few dollars per month
• Some well-known providers: Bluehost, Hostinger, HostGator, InMotion, SiteGround

VPS Database HostingVPS database hosting involves installing databases on a Virtual Private Server. This provides dedicated server space and resources for users to set up their preferred databases. Compared to traditional web hosting, it can result in greater flexibility, performance, and scalability. At the same time, users must configure the environment and manage database updates themselves.

• Cost range: From a few dollars per month to dozens of dollars per month, depending on how large the VPS is
• Some well-known providers: DigitalOcean, InterServer, IONOS, Linode, Vultr, Hetzner

Cloud Database HostingCloud database hosting consists of storing databases on virtual servers within a cloud infrastructure, which guarantees high scalability, availability, and agility. Users can easily adjust resources according to changing needs, databases can be accessed from anywhere, and the provider takes care of maintenance, backups, and security for you.

• Cost range: From a few dollars a month to thousands of dollars a month, depending on usage
• Some well-known providers: Amazon Web Services RDS, Microsoft Azure, Google Cloud SQL

Serverless Database HostingServerless database hosting is a type of cloud database hosting where users do not have to worry about the underlying infrastructure. The reason is that the cloud provider automatically handles resource allocation and scaling. This simplifies database management, helping developers focus on application development. Users are billed only for the actual usage.

• Cost range: From a few dollars per month to dozens of dollars per month, based on resources consumed
• Some well-known providers: AWS Lambda, FaunaDB

Top 5 Inexpensive Options for Database MigrationNow that you are familiar with popular approaches and providers for database hosting, you are ready to see more cost-effective solutions.

If you are eager to learn more about these solutions, check out the summary table below:

| Provider | Databases | Storage | Simultaneous Connections | Read Operations | Write Operations | | --- | --- | --- | --- | --- | --- | | Oracle | Oracle Database, Oracle NoSQL | Up to 20 GB, up to 25 GB per table | Up to 30 sessions | Up to 133 million reads per month | Up to 133 million writes per month | | MongoDB | MongoDB | 512 MB | Up to 500 connections | 100 reads per second | 100 writes per second | | Clever Cloud | PostgreSQL, MySQL | 256 MiB, 10 MiB | Up to 5 connections | — | — | | IBM | IBM Cloudant | 1 GB | — | 20 reads per second | 10 writes per second | | Supabase | PostgreSQL | 500 MB | — | — | — |

Disclaimer: Providers may change these limits and even discontinue free options over time.

Let’s explore the top five database migration options to save money!

Oracle Cloud Always Free TierAfter signing up for Oracle Cloud, new users receive $300 in free credit to use on all eligible Oracle Cloud Infrastructure services for up to 30 days. After that, users can continue on an Always Free plan, which provides more than 25 services available for an unlimited period of time.

The Always Free plan includes two Oracle Autonomous Databases. These serverless databases can be used for transaction processing, data warehousing, Oracle APEX application development, and JSON-based application development. Plus, users gain access to an Oracle NoSQL Database supporting up to 3 tables.

🗃️ Databases: Oracle Database and Oracle NoSQL

📋 Plans:

• Oracle Autonomous Database:
  • Maximum of 2 databases
  • Up to 20 GB storage per database
  • Up to 30 simultaneous sessions
• Oracle NoSQL:
  • Up to 133 million reads per month
  • Up to 133 million writes per month
  • Up to 3 tables
  • Up to 25 GB storage per table
  • Read capacity limited to 50 units
  • Write capacity limited to 50 units

👍 Pros:

• 20+ additional Always Free services (e.g., cloud storage, application performance monitoring)
• Free forever
• Seamless upgrade to paid plans at any time
• Both relational and NoSQL options
• $300 cloud credit free trial
• Integration with other Oracle Cloud services

👎 Cons:

• Payment method required
• Proprietary Oracle databases
• Limited to the regions where commercial Oracle Cloud Infrastructure service is available

MongoDB AtlasMongoDB Atlas is a fully managed cloud database service that simplifies deploying, managing, and scaling MongoDB databases. This platform allows users to deploy clusters, which are groups of servers hosting MongoDB databases. These clusters can span multiple cloud regions on providers like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure.

Users can join MongoDB Atlas for free using the Shared plan, ideal for learning and exploring MongoDB in a cloud environment. The plan includes one M0 free cluster per project, offering 512 MB of storage for a basic MongoDB database.

🗃️ Databases: MongoDB

📋 Plans:

• M0 free cluster:
  • 512 MB of space
  • Sort in memory limited to 32 MB
  • No more than 100 read/write operations per second
  • Up to 500 simultaneous connections

👍 Pros:

• Support for AWS, GCP, Microsoft Azure
• Free forever
• One M0 free cluster per project
• Support for MongoDB 7.0
• Easy upgrade to dedicated clusters
• No credit card required to start

👎 Cons:

• Only 512 MB of space available
• No support for backups, primary failover testing, database auditing, network peering connections, and regional outage testing
• Free clusters automatically paused after 60 days of inactivity

Clever CloudClever Cloud is a platform-as-a-service (PaaS) provider that automates the scaling, deployment, and management of applications. Their solutions are optimized for several technological runtimes, including Java, Scala, Node.js, and PHP. The provider also offers software-as-a-service (SaaS) solutions dedicated to DevOps projects.

Clever Cloud comes with a robust infrastructure to help developers focus on coding rather than server management. This includes databases, such as Elasticsearch, Redis, PostgreSQL, and MySQL. For PostgreSQL and MySQL, the company has some interesting free plans, though with certain limitations.

🗃️ Databases: PostgreSQL and MySQL

📋 Plans:

• PostgreSQL:
  • Storage limited to 256 MiB
  • Up to 5 simultaneous connections
  • Shared RAM and vCPUs
• MySQL:
  • Storage limited to 10 MiB
  • Up to 5 simultaneous connections
  • Shared RAM and vCPUs

👍 Pros:

• One-click setup, daily backups, and high availability options for PostgreSQL
• Unlimited number of free databases and applications
• No payment card required
• Free credits for testing the service upon registration
• Cloud services optimized for EU data and privacy regulations

👎 Cons:

• Limited storage and connections
• No logs or metrics
• MySQL plan too limited for real usage

IBM Cloud CloudantYou can create an account for IBM Cloud for free, which gives you access to services like IBM Cloudant. This is a fully managed JSON document database that offers independent serverless scaling of provisioned throughput capacity and storage.

Cloudant is compatible with Apache CouchDB and accessible via an easy-to-use HTTPS API for web, mobile, and IoT applications.

IBM Cloudant is available through a “free forever” Lite plan, limited to 1 GB of storage. Designed for development and evaluation purposes, the Lite plan includes all Cloudant functions but has a limited amount of throughput capacity and data storage.

🗃️ Databases: IBM Cloudant (proprietary NoSQL)

📋 Plans:

• Lite plan:
  • 1 GB storage
  • Up to 20 reads per second
  • Up to 10 writes per second
  • Up to 5 queries per second

👍 Pros:

• 99.99% SLA uptime
• Compliance with GDPR, PCI, SOC2, and ISO 27001
• All JSON documents are stored across three separate availability zones for high availability and disaster recovery
• Easy to upgrade to a Standard plan
• Official SDK available in Java, Node.js, Python, and Go

👎 Cons:

• Proprietary NoSQL database technology
• Maximum JSON document size of only 1 MB
• Only one IBM Cloudant Lite plan instance per IBM Cloud account

SupabaseSupabase is a backend-as-a-service (BaaS) platform. As an open-source Firebase alternative, it provides instant APIs, real-time subscriptions, authentication, data storage, and more. Specifically, it enables users to create new web and mobile application projects with features like a PostgreSQL database, file storage, and authentication out of the box.

Supabase’s FREE plan includes a dedicated PostgreSQL database for up to 2 supported projects. This database offers 500 MB of storage and root access to users, giving them full control over their data.

🗃️ Databases: PostgreSQL

📋 Plans:

• FREE:
  • 500 MB of storage in a dedicated PostgreSQL database
  • 5 GB of bandwidth per month

👍 Pros:

• Dedicated PostgreSQL database with no privilege restrictions
• Possibility to pause and unpause projects at any time
• Open source technology
• Active GitHub community forum and Discord community

👎 Cons:

• Paused after 1 week of inactivity
• Limited to 2 simultaneous active projects

ConclusionIn this guide, you explored different types of database hosting services and analyzed their costs. Fortunately, there are other modern, low-cost, effective solutions you can consider for database migration. Here, you saw the five best free alternatives to popular database hosting solutions to save money. If you are a small business or individual developer, knowing these affordable solutions can help you save significantly!

The post 5 Low-Cost Database Services to Migrate to appeared first on Semaphore.

View Details

OpenAI is a prominent player in the AI market, experiencing explosive growth. Their top-class AI products use Large Language Models (LLMs) to generate human-like outputs. The fastest and most affordable model is the GPT-4o.

OpenAI offers a powerful API that grants access to its advanced LLMs like GPT-4o, GPT-Turbo, GPT-3.5 Turbo, GPT-3, etc., capable of performing various tasks. These include natural language processing, translation, speech-to-text, image prompts to answer, and much more. In this article, we will see the top emerging OpenAI API alternatives.

Benefits of OpenAI APIOpenAI offers powerful language generation capabilities with numerous other benefits:

1. Ability to customize the LLM models: The OpenAI API allows fine-tuning options for better and more customized outputs.
2. Enhanced User Experience: OpenAI API offers a better and more personalized user experience that can engage the user’s experience through different features in its platform.
3. Ease in Scalability: The API offers you the best and most efficient results while allowing you to scale your product with a large amount of data requests.
4. Versatility: The API can be applied to diverse tasks such as creating marketing copy, generating product descriptions, and writing different creative text formats.
5. Improved efficiency and speed: The API is trained such with the large amount of user data that it can execute large and complex problems in minutes and with efficiency.

Why Consider OpenAI Alternatives?While OpenAI API offers you the best, efficient, and accurate results, it may not suit everyone. Let’s see what are some potential drawbacks of OpenAI API:

• Increased Cost: OpenAI API offers different plans with different pricing. The cost can be prohibitive for some users, who have extensive or continuous needs.
• Complexity: Not every developer can utilize the complete and proper usage of OpenAI API due to complexity issues. For effective results, good technical expertise is needed to use it properly.
• Dependence on Data: Because the models effectiveness depends on the data they are trained on, you may see data biases.

Note: As of March 1, 2023, data sent to the OpenAI using their APIs will not be used to train OpenAI models unless you provide the consent explicitly. See more details here.

Benefits of exploring alternativesOpenAI is a leading organization for text generation that offers advanced and powerful models such as GPT-4o. Let’s see the benefits of OpenAI API alternatives:

• Flexibility: A larger range of APIs enables you to locate one that precisely matches your unique requirements.
• Cost Savings: Several substitutes have more reasonable price structures, particularly for those with smaller usage volumes.
• Pay Attention to Particular Needs: Specialized APIs can deliver better performance for particular tasks, providing more value for your specific needs.

Top OpenAI API AlternativesThere are many different options to take into consideration in the field of AI solutions. Here are some of the popular OpenAI API alternatives:

Large Language Models (LLMs) similar to OpenAI’s GPT-3 (Text-based API)Although OpenAI’s GPT-3 and it’s latest versions have been a trailblazing development in large language models (LLMs), several companies have created potent, alternative LLMs.

These models are frequently as good as or better than GPT-3 or GPT-4 in tasks like text generation, summarizing, etc. They also provide their APIs which can be used in your application. Here are some of the best text-based LLMs similar to OpenAI’s GPT-3:

1. Google Cloud AI APIs:Google Cloud AI provides many different APIs for tasks like text generation, translation, conversational AI, etc. These APIs can be easily integrated into your applications, offering best-in-class performance and are accessible through their Google Cloud Platform (GCP).Google Cloud APIs also provide the features like model customization, scalable infrastructure, and integration with other services. Get your API access here.Cost:While the pricing of APIs differs from usage, Google Cloud gives you the feature of “Pay for what you use”. Also, new customers can get $300 as free credits on their platform. Check out here for more details.
2. Anthropic Claude API:Anthropic is a US-based research organization that creates reliable and beneficial AI-based systems. Anthropic AI products include the latest Claude 3, which can perform advanced reasoning, code generation, multilingual processing, and much more just like ChatGPT. Claude is a powerful model known for its strong performance.With Claude API, you can build a strong AI solution and scale it with the custom rate limits given in your chosen plan. Get your API access.Cost:Anthropic Claude API offers different pricing plans:
   1. Claude Instant (Input: $0.80 per million Tokens, Output: $2.40 per million Tokens)
   2. Claude 2.0 (Input: $8 per million Tokens, Output: $24 per million Tokens)
   3. Claude 2.1 (Input: $8 per million Tokens, Output: $24 per million Tokens)While the latest Claude 3 plans are different where all models support the vision and give you 200,000 token context windows. Check out the pricing here.
3. AI21 Labs:AI21 Labs builds the foundation models and advances AI systems for organizations. They have developed innovative tools and APIs that cater to various text-based applications.
   • AI21 Labs offers two different models Foundational models and Task-specific models.
   • They provide you with the best language models, including its flagship model, Jurassic-2 (it is now deprecated and supplanted by Jamba).
   • Their latest top-notch foundation model is Jamba which offers the best quality and performance for your AI needs.AI21 Labs task-specific models are designed to perform specific tasks of the users including the Paraphrase model, Summarize, Text segmentation, RAG contextual answers, and many more.Cost:The AI21 Labs offers two pricing including the “Pay as you Go” and the Custom Plan. The pay-as-you-go plan is best for those who are early in the stage and have restrictions on budget, whereas the custom plan is best for companies who are looking to scale their AI solutions, etc. Check out the complete pricing here.
4. Cohere:Cohere is a leading enterprise AI platform that offers different models for AI text generation. Cohere provides you the flexible and robust enterprise-grade solutions including organization-specific chatbots, customer feedback, etc.Cohere offers 3 products:
   • Command (Retrieval-augmented generation (RAG))EmbedRerankBelow is an accuracy comparison between Cohere RAG and Claude 3 models for advanced AI applications requiring information from documents and enterprise data sources:

Figure 1: Accuracy comparison between Cohere RAG and Claude 3. Source1. These AI solutions help businesses explore, generate, search for, and act upon information in a new way that’s more intuitive and more natural than ever before. Its language models are customizable as per different cases, giving a better performance.Cost:The pricing of Cohere gives you straightforward pricing plans for their two models. The generative models pricing are as follows: * Command R+ model (Input $3.00 per 1M Tokens, Output $15.00 per 1M Tokens) * Command R model (Input $0.50 per 1M Tokens, Output $1.50 per 1M Tokens) * Command R model (Input $2.00 per 1M Tokens, Output $4.00 per 1M Tokens) fine-tuned.The generative models pricing are as follows: * Rerank 3 model ($2.00 per 1K Searches) * Embed 3 model ($0.10 per 1M Tokens) 2. Hugging Face Transformers:Hugging Face provides a library of pre-trained language models, along with tools for fine-tuning and deploying models. Their models are open-source and can be self-hosted or accessed through an API.Cost:Pricing of hugging face transformers is tailored to individual business needs. Check out the pricing of Huggingfacetransformers API.

When comparing the accuracy of various AI models across different metrics, the latest data reveals significant insights (see below).

Figure 2: Accuracy metrics of various AI models. SourceAccording to Stanford’s HELM accuracy measurements, Claude 3 Opus leads in overall accuracy for MMLU (Massive Multitask Language Understanding- a benchmark used to assess the performance of AI models on a diverse set of tasks) All Subjects with an impressive score of 0.846. Following closely is GPT-4o (2024-05-13), then Google’s Gemini 1.5 Pro (001), and many more.

This comparison highlights the competitive edge of newer models like Claude 3 Opus and GPT-4o in general accuracy, while also showcasing specialized strengths in specific domains like Abstract Algebra by Gemini 1.5 Pro.

Specific AI ServicesApart from all-purpose large language models, many companies provide AI services specifically designed for particular jobs like computer vision, speech recognition, natural language processing, and image production.

To illustrate the capabilities of various AI models, here is a detailed comparison of performance metrics for some of the leading AI services:

Figure 3: Performance comparison of leading AI models across various tasks. SourceLet’s see the specific AI services-based API alternatives to OpenAI:

Text-to-Speech1. Amazon Polly API:Amazon Polly API provides high-quality text-to-speech conversion. It offers high-quality generative lifelike speech from text, as long-form, neural, and high-quality voices both in men’s and women’s voice options. Amazon Polly offers several API operations for integration in your application.Features: * Cloud-based solution * Low latency * Multiple language and voice optionsCost:The Amazon Polly API is a pay-per-use model and there are no setup costs in that. The great thing with Polly is that you can start with your small application and scale later on with ease. See more pricing details for Amazon Polly API.

Computer Vision1. Microsoft Azure Cognitive Services:Azure AI Vision offers innovative computer vision capabilities. With Azure AI Vision you can perform tasks like image recognition, object detection, facial recognition, and text extraction with optical character recognition (OCR). Features: * Facial recognition * Video streaming using Spatial analysis * Image and object detectionCost:Microsoft Azure AI Vision API is a Pay for only what you use with no upfront costs. The pricing model of Azure AI Vision is a pay-as-you-go and calculated based on the number of transactions consumed. See more pricing details.New users can get $200 credits FREE to be used within 30 days. Check it out here. 2. Clarifai API:Clarifai API offers high-quality AI models based on deep learning. The Clarifai API provides a human-like interpretation of video, image, text, and audio. Clarifai API provides customizable computer vision models and a wide range of applications.Features: * Image recognition using their Pre-trained models * Video Analysis using Deep learning models * OCR (Image to Text) and Image labelingCost:Clarifai API offers a user-based pricing model and there are no setup costs in that. You can find their different pricing plans and API costs here.

Machine Learning (ML) and Natural Language Processing (NLP)1. IBM Watson:IBM Watson is a comprehensive platform providing various AI capabilities, including sentiment analysis, question-answering, and machine learning tools. IBM Watson offers different AI models including Watson Natural Language Understanding, Watson Speech to Text and vice versa, watsonx code assistant, etc.IBM Watson APIs facilitate the development of enterprise-class apps that integrate natural language processing capabilities into any hybrid multi-cloud setup. Check out the complete details of IBM Watson APIs here.Features: * Natural language understanding * Best ML models * Language TranslationCost:IBM Watson offers flexible pricing plans depending on the chosen service, including a free tier. Check out the IBM custom products costs here. 2. Google Cloud Natural Language API:Google Cloud Natural Language API is the best natural language AI model that helps you analyze and derive insights from unstructured text. Google Natural Language API allows the customization of the models to classify, extract, and detect sentiment with minimum effort. It also allows us to analyze multi-language text, content classification, and much more.Features: * Sentiment Analysis * Object Recognition * Syntax Analysis (including tokenization and part-of-speech tagging and much more)Cost:Google Cloud Natural Language API offers the pay for what you use and you may also get up to $300 in free credits. Check out the detailed pricing for using the Google Cloud Natural Language API here.

Automatic Speech Recognition1. Google Cloud Speech-to-Text API:Google Cloud Speech-to-Text API allows to conversion of audio into text transcriptions. Using Google’s AI Speech-to-text API, you can easily integrate speech recognition into your applications. It supports the speech in two formats including microphone and file upload.Features: * Advanced speech AI * Transcribe audio using Pre-trained models * Caption videos using AICost:The pricing of Google Cloud Speech-to-Text API is based on the API version, channels, etc methods. For the API version “Speech-to-Text V1 API” pricing is $0.024 per min whereas for “Speech-to-Text V2 API” the cost is $0.016 per min.You may also get up to $300 in free credits here. Check out the detailed pricing for using the Google Cloud Speech-to-Text API here. 2. AssemblyAI APIWith AssemblyAI API you can build the best in industry AI speech models. Assembly API provides high accuracy in speech-to-text and gives support for multiple audio formats including multi-language. The API is easy to integrate and has strong developer support. AssemblyAI has three products: * Speech-to-Text * Streaming Speech-to-Text * Speech UnderstandingFeatures: * Sentiment Analysis * Object Recognition * Syntax Analysis (including tokenization and part-of-speech tagging and much more)Cost:AssemblyAI offers three different pricing plans including the Free tier, pay-as-you-go (start as low as $0.12 per hour for Speech-to-Text), and custom plans according to needs. Check out the detailed pricing here.

Image Generation APIs1. Amazon Titan APIWith the Amazon Titan Image generation API, you can generate high-efficiency images from the text input. Titan API can easily generate realistic, ready-to-use images with high quality in seconds just using natural language prompts. It also allows image editing and image variations.The advanced AI model understands complex instructions with multiple objects and returns studio-quality images that can be used in e-commerce, advertisements, etc. Check out the Amazon Titan Image Generator Demo.Features: * Text-to-image generation * Customization and fine-tuning options * Scalability and IntegrationCost:Amazon Titan API offers different pricing models including One-Demand (pay-as-you-go), batch, etc. Check out the complete pricing details of the Amazon Bedrock Titan API here. 2. DeepAI APIWith the DeepAI AI Image generator API, you can generate images using the text prompts. With the DeepAI API, you can generate three types of images including Standard, HD, and Genius. DeepAI generates high-quality realistic images using text prompts.Features: * Versatile Image Generation * API Customization * Monthly SubscriptionCost:DeepAI Pro is something they offer as a monthly subscription with access to all of their 100+ generative tools and features. Check out the complete pricing details of the DeepAI here. 3. Midjourney APIMidjourney is a cutting-edge tool for AI image creation. The Midjourney API empowers you to seamlessly incorporate Midjourney into your applications, allowing you to generate unique and realistic images based on text prompts.Features: * Realistic image generation * Fine-tuning and customization * Developer community supportCost:Midjourney API offers four different monthly and year plan subscriptions. It includes the Basic plan ($10 per month), the Standard plan ($30 per month), the Pro plan ($60 per month) and the Mega plan ($120 per month). Check out the complete details of Midjourney API pricing.

Choosing the Right AlternativeWhen choosing an alternative to the OpenAI API, consider the following factors:

• Functionality: Assess the specific capabilities and use cases supported by each alternative to ensure they meet your project’s requirements.
• Pricing: Compare the pricing models and overall costs of different models to find the most cost-effective one for your budget and usage patterns.
• Ease of Use: Consider the ease of integration, documentation, and support each provider offers, as it can significantly impact development time and effort.
• Privacy and Security: Evaluate the privacy and security measures implemented by each provider, especially if your project involves sensitive or regulated data.
• Project Goals: Ensure that the alternative aligns with your project’s specific goals.

ConclusionIn this article, we have seen the best OpenAI API alternatives. While OpenAI’s API has raised the bar for AI services, there are a lot of other options with more features and advantages to consider as OpenAI alternatives. Businesses and developers can find solutions that better suit their particular needs by looking at these options, accomplishing so may result in more flexibility and cost savings.

A perfect tool for every purpose can be found thanks to the wide range of available AI APIs that are currently available, whether it’s for text generation, image generation, speech recognition, computer vision, natural language processing (NLP), or any other application.

The post OpenAI API Alternatives appeared first on Semaphore.

View Details

When building a software product, whether a framework, library or any interesting piece of technology, having a well-designed documentation website that provides a great user experience is non-negotiable. No matter how exceptional the product itself may be, if the documentation falls short, it can severely impact the adoption and usability of that product. Too often, users complain about wasting time searching for answers, resorting to external resources, or expressing frustration over the minimal effort put into official documentation.

For developers or organizations who have invested countless hours into building a product, such critiques can be disheartening, but they highlight an important reality: if your documentation website is poorly built, it creates a bad impression of the product’s quality.

However, building a software documentation website from scratch can be daunting. It requires great effort and keen attention to detail. Using general-purpose tools like Next.js is certainly a viable option, but it necessitates manually handling aspects such as content management, SEO, accessibility, versioning, and other technical complexities. Dealing with these complexities can divert attention from the core goal of creating comprehensive software documentation.

Over the years, several specialized tools have emerged to address these challenges. One such tool that has gained popularity is Docusaurus. Docusaurus makes building software documentation extremely easy by handling the technical heavy lifting, allowing you to focus more on creating high-quality content instead of worrying about the underlying infrastructure. In this article, we will learn how to use Docusaurus to build visually appealing documentation that is not only functional but also a pleasure to use.

What is Docusaurus?Docusaurus is an open-source tool for building documentation websites. It was created by engineers at Meta to make creating and maintaining documentation a delightful experience. The idea behind Docusaurus is to allow you to write documentation using Markdown and React-based MDX. This combination makes it easy to create both regular documentation pages and interactive content. One of the biggest advantages of Docusaurus is that it is built on top of React, which means you can leverage your existing React knowledge and write custom React components. Many popular companies, such as Algolia, Redis Labs Developer Site, Atlas, Jest, and React Native, to mention just a few, have adopted Docusaurus for their documentation due to the excellent developer experience it provides.

Why Choose Docusaurus? A Comparison with AlternativesDocusaurus is often the popular choice for building documentation websites, but it’s not the only option available. Let’s compare Docusaurus with some popular alternatives:

| Feature | Docusaurus | GitBook | MkDocs | VuePress | Jekyll | Sphinx | | --- | --- | --- | --- | --- | --- | --- | | Base Technology | React | JavaScript | Python | Vue | Ruby | Python | | Customization | High | Moderate | Moderate | High | High | High | | Learning Curve | Moderate | Low | Low | Moderate | Moderate | Steep | | Versioning | Built-in | Limited | Plugin | Plugin | Plugin | Built-in | | Internationalization | Built-in | Built-in | Plugin | Plugin | Plugin | Built-in | | Performance | High | Moderate | High | High | Moderate | High | | MDX Support | Yes | No | No | No | No | No | | Search | Built-in | Built-in | Plugin | Plugin | Plugin | Built-in | | Output Formats | Web | Web, PDF | Web | Web | Web | Web, PDF, ePub | | Best For | Large web docs, React devs | Quick setup, non-devs | Small to medium projects | Vue devs | Static sites | Python projects |

Each tool has its strengths, but Docusaurus shines when you need a modern, customizable documentation website with out-of-the-box features like versioning and internationalization. Its React foundation makes it particularly appealing for teams already working with React-based projects.

PrerequisitesTo ensure you can follow along effectively, this tutorial assumes you have the following:

• Node.js installed
• Basic knowledge of React
• Basic understanding of Markdown syntax

Getting StartedDocusaurus streamlines the process of setting up a documentation website, which allows you to go from concept to deployment quickly. In this section, we will initialize a new Docusaurus project, explore its structure, and get it running locally.

Installing DocusaurusDocusaurus provides a useful CLI tool that scaffolds a new project with sensible defaults. To initialize a new project, open your terminal and run:

npx create-docusaurus@latest my-docs classic This will create a new Docusaurus project named my-docs, but you can use any name you prefer. The classic flag specifies that this project will use the classic template, which includes essential features and configurations for a documentation site. During setup, you will be prompted to choose:

• Javascript or Typescript (I’ll be using Javascript for the purposes of this tutorial)
• Optional GitLab/GitHub repository creation

Alternatively, you can use any package manager of your choice like npm, yarn, or pnpm straight from the get-go:

npm init docusaurus# oryarn init docusaurus# orpnpm init docusaurus Project StructureLet’s navigate into the newly created Docusaurus project and get familiarised with its structure:

cd my-docs You will see something like:

my-docs├── blog├── docs│ ├── intro.md│ └── tutorial-basics│ ├── congratulations.md│ ├── create-a-blog-post.md│ ├── create-a-document.md│ ├── create-a-page.md│ ├── deploy-your-site.md│ ├── markdown-features.md│ └── thank-you.md├── package.json├── src│ ├── components│ ├── css│ └── pages├── static├── .gitignore├── barbel.config.js├── package.json├── package-lock.json└── docusaurus.config.js Key directories:

• blog/: This directory contains blog posts written in Markdown or mdx.
• docs/: This is where you we will place your documentation files in Markdown or mdx format.
• src/: Here, you can create pages and customize components, layouts, and other React-related code for your site.
• static/: This directory is for static assets like images, fonts, etc.
• docusaurus.config.js: This is the main configuration file for your Docusaurus site.
• sidebars.js: This file is responsible for auto-generating and customizing the sidebar.

Running LocallyFirst, run npm install to install all the required dependencies, and then start your development server:

npm run start This will compile all the Markdown and React files, start a local development server (typically at http://localhost:3000), and enable hot reloading for instant updates. You should see the default Docusaurus site running in your browser.

Understanding the Config FileWe will begin by exploring the docusaurus.config.js. This file acts as a control center for your Docusaurus project where you can tweak the site’s behavior, appearance, and functionality. Let’s dissect this config file and understand how each part shapes your documentation’s identity and user experience.

Open docusaurus.config.js in your editor. You will see a structure like this:

const config = { title: "My Site", tagline: "Dinosaurs are cool", favicon: "img/favicon.ico", url: "https://your-docusaurus-site.example.com", baseUrl: "/", organizationName: "facebook", projectName: "docusaurus", onBrokenLinks: "throw", onBrokenMarkdownLinks: "warn", i18n: { defaultLocale: "en", locales: ["en"], }, //...};export default config; Core Fields:* title: The title of your Docusaurus site. * tagline: A short tagline or description for your site. * favicon: The path to the favicon icon for your site. * url: The URL where your site will be hosted. * baseUrl: The base URL path for your site (typically /). * organizationName and projectName: These are used for certain Docusaurus features like the edit button. * onBrokenLinks and onBrokenMarkdownLinks: These control how Docusaurus handles broken links. * i18n: This object configures internationalization (i18n) settings like the default locale and supported locales.

Theme ConfigurationThe themeConfig object is where you define your site’s look and feel:

const config = { //... themeConfig: { image: "img/docusaurus-social-card.jpg", navbar: { title: "My Site", logo: { alt: "My Site Logo", src: "img/logo.svg", }, items: [ { type: "docSidebar", sidebarId: "tutorialSidebar", position: "left", label: "Tutorial", }, { to: "/blog", label: "Blog", position: "left" }, { href: "https://github.com/facebook/docusaurus", label: "GitHub", position: "right", }, ], }, footer: { style: "dark", links: [ { title: "Docs", items: [ { label: "Tutorial", to: "/docs/intro", }, ], }, { title: "Community", items: [ { label: "Stack Overflow", href: "https://stackoverflow.com/questions/tagged/docusaurus", }, { label: "Discord", href: "https://discordapp.com/invite/docusaurus", }, { label: "Twitter", href: "https://twitter.com/docusaurus", }, ], }, { title: "More", items: [ { label: "Blog", to: "/blog", }, { label: "GitHub", href: "https://github.com/facebook/docusaurus", }, ], }, ], copyright: `Copyright © ${new Date().getFullYear()} My Project, Inc. Built with Docusaurus.`, }, prism: { theme: prismThemes.github, darkTheme: prismThemes.dracula, }, }, //...}; * navbar: Define logo, title, and navigation items. * footer: Set style, links, and copyright notice. * prism: Choose a syntax highlighting theme.

Presets and ThemesDocusaurus uses presets to bundle plugins and themes:

const config = { //... presets: [ [ "classic", { docs: { sidebarPath: "./sidebars.js", editUrl: "link/to/edit/page", }, blog: { showReadingTime: true, editUrl: "link/to/edit/page", }, theme: { customCss: "./src/css/custom.css", }, }, ], ], //...}; * docs: Set path and sidebar configuration. * blog: Configure blog features. * theme: Apply custom CSS for fine-tuned styling.

This is a quick rundown of the docusaurus.config.js file. The great thing about this file is that you can customize various aspects of your documentation by modifying it, without needing to touch other files or write additional code. For example, If you change the title and tagline to something else, it will update the homepage and other parts where these values are used. The following example demonstrates this:

const config = { title: "Semaphore", tagline: "Build, Test, Deploy - At Lightning Speed", //...}; This will update the homepage title and tagline without having to touch src/pages/index.js directly.

Creating PagesDocusaurus allows you to create standalone pages in the src/pages directory. These pages are different from documentation pages and can be used for custom content such as a homepage, about page, or any other static content you need. To create a new page, navigate to the src/pages folder. If you need to create an about page, for example, you would create about.jsx file inside that folder:

touch src/pages/about.jsx Add content to about.jsx file:

``` import React from "react";import Layout from "@theme/Layout";function About() { return ( About Us Semaphore CI is a cloud-based continuous integration and delivery platform that accelerates software development. We help teams build test, and deploy code faster and more reliably.

   );}export default About;

``` Now visit: http://localhost:3000/about

This code defines a simple React component for the about page. The layout component from Docusaurus provides the site’s header, footer, and other layout features. While it is convenient to use Javascript/JSX to create pages, you also have the option to use Markdown or React MDX.

Add the Page to the NavbarTo make about page accessible via the navigation bar, you need to update the docusaurus.config.js file in the root directory.

const config = { //... themeConfig: { image: "img/docusaurus-social-card.jpg", navbar: { title: "My Site", logo: { alt: "My Site Logo", src: "img/logo.svg", }, items: [ { type: "docSidebar", sidebarId: "tutorialSidebar", position: "left", label: "Tutorial", }, { to: "/blog", label: "Blog", position: "left" }, // Add new Navlink here { to: "about", label: "About", position: "left" }, { href: "https://github.com/facebook/docusaurus", label: "GitHub", position: "right", }, ], }, //... }, //...}; This configuration adds a link to the About page in the navigation bar. Check the browser to see the change.

Creating documentation pageThe documentation content in Docusaurus is typically located in the docs directory of your project. Let’s start fresh by deleting the default files in the docs folder. Next, we will rename the Tutorial navbar item to Docs to better represent a real documentation website. Open the docusaurus.config.js file and locate the themeConfig.navbar.items section. Update the label field of the docSidebar item from Tutorial to Docs:

const config = { themeConfig: { navbar: { items: [ // ... { type: "docSidebar", sidebarId: "tutorialSidebar", position: "left", label: "Docs", }, // ... ], }, },}; To create a new documentation page, make sure you are in docs folder. Create a new Markdown or React MDX file. For this example, let’s create a file named welcome.mdx:

touch docs/welcome.mdx Open welcome.mdx in your text editor and add the following content:

---id: welcometitle: Welcomedescription: This is a description for the welcome pageslug: /welcome---**# Welcome to my Doc**This is my first documentation page in Docusaurus. The new welcome page link would appear in the documentation sidebar. You can also navigate to localhost:3000/docs/welcome in the browser to access the page.

Understanding the structure of a Documentation PageDocusaurus documentation pages consist of two main parts:

• Front Matter: The top section enclosed in triple dashes (---). It contains metadata like the unique id, title, description, slug, and other details about the page. This metadata guides how Docusaurus generates the sidebar, page titles, URLs, and more. Think of it like the section in HTML. The metadata is not visible in the rendered documentation page but is used by Docusaurus to generate the sidebar navigation, page titles, URLs, and other features.
• Main Content: This is where you write the actual documentation using Markdown syntax. Here, you can structure the content with headings, code blocks, images, and take advantage of the full expressiveness of Markdown.

Customizing the Documentation PageYou can customize the behavior and appearance of your documentation pages using various front-matter fields. For example, to change the order of a page in the sidebar, you can use the sidebar_position field:

---id: welcometitle: Welcomedescription: This is a description for the welcome pageslug: /welcomesidebar_position: 1---**# Your Markdown content goes here** Setting sidebar_position: 1 will move the page to the first position in the sidebar. Other accepted front matter fields include:

• sidebar_label: The label displayed in the sidebar navigation for this document.
• hide_title: A boolean flag to hide the title at the top of the document page.
• pagination_label: A label for the pagination navigation at the bottom of the document page.
• custom_edit_url: A custom URL for the “Edit this page” link.
• keywords: An array of keywords related to the document, used for search engine optimization.
• image: A relative path or URL to an image that can be used as a preview or social media thumbnail for the document.
• hide_table_of_contents: A boolean flag to hide the table of contents for the document.
• toc_min_heading_level: An integer value (1-6) that specifies the minimum heading level to be included in the table of contents.
• toc_max_heading_level: An integer value (1-6) that specifies the maximum heading level to be included in the table of contents.
• pagination_next: A label for the “Next” pagination link.
• pagination_prev: A label for the “Previous” pagination link.

Grouping Documentation PagesYou can group pages in Docusaurus by creating a subfolder inside the docs directory. Let’s say you want to create a “Getting Started” category with pages for “Installation” and “Integration”. First, create the subfolder inside docs directory:

mkdir docs/getting-started Inside the getting-started folder, create a JSON file named _category_.json. This file defines the sidebar navigation structure for the category. Add the following:

{ "label": "Getting Started", "position": 1, "link": { "type": "generated-index" }} The label specifies the category name, and position determines its order in the sidebar. The link field creates an index page for the category, listing all its pages.

Adding Pages to the CategoryNext, create the pages for the category:

touch docs/getting-started/installation.md docs/getting-started/integration.md Add content to installation.md page with appropriate front matter:

---id: installationtitle: Installationsidebar_position: 1---**# Installation**Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Add content to integration.md:

---id: integrationtitle: Integrationsidebar_position: 2---**# Integration**Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Now, these pages will be grouped under the “Getting Started” category in the sidebar, with the index page listing them.

Using React in DocusaurusDocusaurus allows you to embed React components directly in your documentation pages, giving you the flexibility to create interactive and dynamic content. This is possible thanks to MDX (Markdown with JSX), which allows you to use JSX within your Markdown files. This feature is particularly useful when you want to showcase live examples, abstract away repetitive code, or add complex interactivity to your docs. To use React components in your documentation pages, ensure your documentation file format is .mdx instead of .md. For this example, we are going to create a basic component named ColorBox. In your project’s root directory, navigate to the src/componentsfolder (create it if it doesn’t exist) and create a .js or .jsx component:

mkdir -p src/componentstouch src/components/ColorBox.jsx Add content to the file:

import React from 'react';const ColorBox = ({ color }) => ( `);export default ColorBox;` You can then use this component in your MDX file. This example uses the previously created welcome.mdx file for demonstration:

---id: welcometitle: Welcomedescription: This is a description for the welcome pageslug: /welcome---import ColorBox from '@site/src/components/ColorBox';**# Welcome to my Doc**This is my first documentation page in Docusaurus. Note: The @site alias in the import statement is a special path alias in Docusaurus that points to your project’s root directory.

In the browser, you should see the custom component render along with Markdown content:

This approach allows you to create rich and interactive documentation that goes beyond static text and images. You can embed live code editors, interactive diagrams, or even small applications directly in your documentation pages.

Pro Tip: Although React components can greatly enhance your documentation, use them judiciously. For simple content, plain Markdown is often more maintainable and performs better. Reserve React components for interactive elements or complex visualizations that can’t be achieved with standard Markdown.

Versioning DocumentationDocusaurus has built-in support for versioning documentation which allows you to maintain multiple versions of your documentation and provide version-specific content to your users. To enable versioning, you first need to configure your Docusaurus project to support it.

Open docusaurus.config.js and update it to include versioning settings. Add the following to the docs field in the config file:

const config = { //... presets: [ [ "classic", { docs: { // .... lastVersion: "current", versions: { current: { label: "Next", path: "next", }, }, }, //... }, ], ], //...}; This configuration sets up versioning with a Next version, which is the default label for the current version of your documentation.

To create an initial version of your documentation, run the following command:

npm run docusaurus docs:version 1.0.0 This will create a version-1.0.0 directory inside the new versioned_docs directory and a versions.json file in the project directory. You sould have folder structure like this:

my-docs/├── docs/│ └── ... # Current documentation files├── versioned_docs/│ └── version-1.0.0/│ └── ... # Copied documentation files for version 1.0.0├── versions.json # Metadata file for versioning├── ... Creating New VersionsTo add more versions of your documentation, run the docs:version command with the new version number:

npm run docusaurus docs:version 1.1.0 This will create a new directory version-1.1.0 inside the versioned_docs folder, and the documentation files from the docs directory will be copied into it. The new version will also be added to the versions.json file. After creating a new version, you should have an updated folder structure like this:

my-docs/├── docs/│ └── ... # Current documentation files├── versioned_docs/│ ├── version-1.0.0/│ │ └── ... # Documentation files for version 1.0.0│ └── version-1.1.0/│ └── ... # Documentation files for version 1.1.0├── versions.json├── ... You can continue updating the documentation files in the docs directory, and when you are ready to create a new version, simply run the docusaurus docs:version command again with the new version number.

Managing VersionsThe versions.json file in the root directory contains metadata about the different versions of your documentation. It looks like this:

["1.0.0", "1.1.0"] You can manually edit this file to add or remove versions as needed.

Adding Version Dropdown to NavbarAfter setting up versioning, you can update the navbar in docusaurus.config.js to display a version dropdown menu. Open the config file, in config.themeConfig.items, add the following:

const config = { //... themeConfig: { //... items: [ // ... { type: "docsVersionDropdown", position: "right", }, ], }, //...}; This will show a version dropdown menu in the navigation bar.

Pro Tip: If your documentation changes infrequently, versioning may introduce unnecessary complexity. In most cases, maintaining a single, up-to-date version of your documentation is simpler and more effective.

Exploring PluginsDocusaurus has a thriving ecosystem of plugins that can extend the functionality of your documentation site. Let’s look at an example of integrating Translation and Internationalization plugins in Docusaurus.

Translation and InternationalizationDocusaurus has built-in support for internationalization (i18n) that allows you to create a multi-language documentation website. This feature enables you to reach a global audience by providing translated content in multiple languages. Let’s set up and manage translations in Docusaurus project.

Configuring LocalesThe first step is to configure the locales (languages) you want to support in your documentation site. Open the docusaurus.config.js file and locate the i18n object. By default, it should look like this:

i18n: { defaultLocale: 'en', locales: ['en'], }, To add a new locale, for example, French (fr), update the locales array:

i18n: { defaultLocale: 'en', locales: ['en', 'fr'], }, Translating ContentAfter configuring the locales, you can start translating your content. Docusaurus expects translations to be placed in a specific directory structure within the project-root/i18n folder. Create a directory named i18n in the root folder:

mkdir -p i18n Docusaurus provides a CLI that you can use to manage translations. To extract the translatable content from your Markdown and React components, run the following command

npm run write-translations This command, by default, will generate only the English (en) locale directory along with translatable JSON files inside the i18n directory. To instruct the command to generate a French locale or your desired locale, you will have to specify it by providing the --locale flag. For example, to generate the French (fr) locale, run:

npm run write-translations -- --locale fr This will generate the fr locale directory containing translatable files inside the i18n directory. You should have a folder structure like this:

my-docs/└── i18n ├── en │ ├── code.json │ ├── docusaurus-plugin-content-blog │ │ └── options.json │ ├── docusaurus-plugin-content-docs │ │ ├── current.json │ │ ├── version-1.0.0.json │ │ └── version-1.1.0.json │ └── docusaurus-theme-classic │ ├── footer.json │ └── navbar.json ├── fr │ ├── code.json │ ├── docusaurus-plugin-content-blog │ │ └── options.json │ ├── docusaurus-plugin-content-docs │ │ ├── current │ │ │ ├── version-1.0.0 │ │ ├── current.json │ │ ├── version-1.0.0.json │ │ └── version-1.1.0.json │ └── docusaurus-theme-classic │ ├── footer.json │ └── navbar.json //.... Every JSON file in the locale represents a specific aspect of your documentation where you can manually translate the content. For instance, current.json is responsible for translating general text content and UI elements like the site title, description, and theme labels. The navbar.json and footer.json in the i18n/fr/docusaurus-theme-classic directory are where you translate the navbarand footer content to French, respectively. You can explore all the JSON files and translate as many aspects as you want to create a fully localized documentation website.

To translate your documentation content, you have to copy your documentation content from the docs files to i18n/fr/docusaurus-plugin-content-docs/current/. Copy all the files inside the docs directory:

mkdir -p i18n/fr/docusaurus-plugin-content-docs/current cp -r docs/* i18n/fr/docusaurus-plugin-content-docs/current/ After copying, you have to manually translate the content to your desired language, which in this case is French. The structure for the multilingual setup should look like this:

my-docs/└── i18n ├── en │ ├── code.json │ ├── docusaurus-plugin-content-blog │ │ └── options.json │ ├── docusaurus-plugin-content-docs │ │ ├── current.json │ │ ├── version-1.0.0.json │ │ └── version-1.1.0.json │ └── docusaurus-theme-classic │ ├── footer.json │ └── navbar.json ├── fr │ ├── code.json │ ├── docusaurus-plugin-content-blog │ │ └── options.json │ ├── docusaurus-plugin-content-docs │ │ ├── current │ │ │ ├── getting-started │ │ │ │ ├── installation.md │ │ │ │ ├── integration.md │ │ │ ├── welcome.mdx │ │ │ ├── version-1.0.0 │ │ ├── current.json │ │ ├── version-1.0.0.json │ │ └── version-1.1.0.json │ └── docusaurus-theme-classic │ ├── footer.json │ └── navbar.json For transaltion to take effect, you will need to restart your server and add the --locale flag.

npm run start -- --locale fr Now, when you refresh the browser, you should see something like this when you visit localhost:3000/fr/docs/next/welcome:

Enabling the Language DropdownAfter configuring the locales and providing translations, you can enable a language dropdown in your site’s navigation bar. Open docusaurus.config.js file and locate the themeConfig.navbar.items array. Add the following item:

const config = { themeConfig: { navbar: { items: [ // ... { type: "localeDropdown", position: "right", }, ], }, },}; This will add a language dropdown to the right side of the navigation bar, allowing users to switch between the configured locales.

Routing and URL StructureNote: Internationalization works a little differently in the dev environment compared to production. In production, you can seamlessly switch between the provided languages without any issues. However, in the development environment, you need to specify the locale when starting the server, and it only supports one language at a time. This means if you start your dev server in the French locale and switch to, say, English, you will get a 404 error. In production, this will not be an issue as Docusaurus automatically handles the routing and URL structure for your translated content. When a user selects a different locale from the language dropdown, Docusaurus will load the corresponding translated content and update the URL to reflect the selected locale. For example, if a user visits the /docs/welcome URL on the English (en) site, and then switches to the French (fr) locale, Docusaurus will load the French translation of the welcome.mdxfile and update the URL to /fr/docs/welcome.

More about about PluginsEarlier, we learned how to leverage the localization plugin, but that is just a small taste of what plugins can do for your Docusaurus site. There are tons of useful plugins that make things easier. Let’s say you need to integrate Google Analytics into your project. Here is how:

Install the PluginFirst, install the required plugin by running this command:

npm install --save @docusaurus/plugin-google-gtag With the plugin installed, you can configure it in your docusaurus.config.js file. Look for the presets array and find the presets entry. Then add this gtag config inside it:

const config = { //... presets: [ [ 'classic', /** @type {import('@docusaurus/preset-classic').Options} */ ({ gtag: { trackingID: 'G-999X9XX9XX', anonymizeIP: true, }, docs: { sidebarPath: './sidebars.js', lastVersion: 'current', versions: { current: { label: 'Next', path: 'next', }, }, }, //... }), ], ], //...}; Make sure to swap out G-999X9XX9XX with your actual Google Analytics tracking ID. The anonymizeIP option helps with privacy by anonymizing visitor IP addresses before sending data. Google Tag is disabled in the development environment. This behavior is intentional to prevent unnecessary analytics data from being sent while you’re developing and testing your site locally. Google Tag will be enabled in the production environment and your site should be able to collect analytics data. You can use tagassistant.google to confirm if your Google Analytics implementation is working correctly using your production link.

You can explore more plugins from Docusaurus official website to extend the capabilities and functionality of your documentation website.

Deploying Your Docusaurus WebsiteAfter building your documentation website, there is nothing more satisfying than seeing it live and running. Docusaurus provides a CLI to streamline deployment, making the process straightforward. You can also deploy your site to cloud services like Vercel or Netlify, which offer additional features and ease of use.

Using Docusaurus CLI for DeploymentDocusaurus includes a built-in command-line tool that simplifies the deployment process. To deploy your site using the Docusaurus CLI:

Ensure that your site is built and ready for deployment. Run the following command to create a production-ready build:

npm run build This will generate static files in the build directory, which can be served by any static site hosting service.

To ensure everything works as intended, test your build locally by running the following command:

npm run serve This will start a local server and you can view your site by navigating to the provided URL. Make sure to check all pages and features to verify that they are functioning correctly.

Deploying to Cloud ServicesFor more flexibility, deploy your Docusaurus site to cloud services like Vercel or Netlify, which offer seamless integration with Git and automated deployment workflows. Follow their setup guides:

• Deploy to Vercel
• Deploy to Netlify

Pro Tip: Fix broken links in your Docusaurus project to prevent deployment failures.Earlier in this tutorial, we deleted all default files and folders in the docs directory. This may have created broken links, particularly references to docs/intro. To ensure smooth deployment:

1. Open docusaurus.config.js
2. Navigate to config.themeConfig.footer.links.items
3. Delete or Replace /docs/intro with a working link or use the relative path /.

const config = { //... themeConfig: { footer: { style: "dark", links: [ { title: "Docs", items: [ { label: "Tutorial", to: "/docs/intro", // Resolve this broken link }, ], }, //... ], }, }, //...}; Repeat this process for src/pages/index.js, updating any similar references.

Wrapping UpWe have covered a lot of ground in this guide. We touched on key concepts such as initializing a new Docusaurus project, creating documentation, versioning, integrating plugins, deploying a Docusaurus website, and more. Building a documentation website is not a simple task, but tools like Docusaurus take a lot of that complexity out of the equation by providing streamlined workflows and essential features out of the box. While this guide aims to provide a solid foundation, there is still much more to explore with Docusaurus. There are limitless possibilities for customization through Docusaurus’ extensive ecosystem of plugins and themes.

Feel free to clone my working repository in parallel to the fresh, walk-thru provided here. Please do reach out to me for any questions/clarifications or bugfixes!

The post Using Docusaurus to Build A Modern Documentation Website appeared first on Semaphore.

View Details

Conventional data analysis workflows can be cumbersome and require keeping up with the exponential volume and complexity of modern data sources, which may result in significant hurdles, such as being time-consuming, error-prone, and requiring specialized expertise. Automating these processes can significantly reduce the time and resources needed for data analysis.

Retrieval-augmented generation (RAG)-enabled large language models (LLMs) offer a trivial solution to the challenges of conventional data analysis workflows. By combining the capabilities of natural language processing (NLP) with the ability to retrieve and integrate relevant information from external sources into its response, RAG-enabled LLMs can access and process vast amounts of structured, semi-structured, and unstructured data. Organizations can automate various data analysis tasks by leveraging RAG-enabled LLMs, including summarising data trends, identifying correlations and patterns, and detecting anomalies. With RAG-enabled LLMs, the time and effort required for data analysis are significantly reduced while accuracy and efficiency increase.

Here, you’ll explore the possibilities of using RAG-enabled LLMs to automate data analysis and how it transforms the way we work with data.

Prerequisites Understanding of Python, Data Analysis, and How Large Language Models Work. * An OpenAI account and API key for accessing their LLMs * You can find the data required to follow along on the GitHub repository. * Development Environment*: Create and activate a virtual environment with Python in your workspace directory. Then, proceed to install the required packages.

python -m venv .venv && source .venv/bin/activatepip3 install pypdf python-dotenv pandas beautifulsoup4 llama-index==0.9.48 What are RAG-Enabled LLMs?Large language models (LLMs) have become robust tools for processing natural language. Due to continuous increases in their context window, they can fluently comprehend, produce, and modify human language.

The architecture of LLMs is a crucial factor in their versatility. It allows data from external sources to be seamlessly integrated, making LLMs a base or kernel for building on top of.

Connection to external data enables the development of RAG-enabled LLMs, a prevalent paradigm for fitting external data into LLMs.

Retrieval-Augmented Generation (RAG)One primary reason for RAG is that LLMs do not have access to all or most of the data you might care about, such as private data or data generated after training the LLM.

RAG combines the strengths of both generative and retrieval-based models. Its architecture consists of multiple components, including Query transformations, Routing, Query construction, Indexing, Retrieval, and Generation. The focus here will be on the last three:

• Indexing: This involves creating an index or a searchable database of a collection of related data or documents after chunking and embedding them. The indexing mechanism enhances the model’s retrieval of accurate information.
• Retrieval: Unlike traditional LLMs, RAG incorporates a retrieval mechanism responsible for retrieving relevant information from external knowledge sources, such as text databases or pre-indexed corpora. This mechanism serves as additional context for the generative component.
• Generation: This component functions similarly to conventional LLMs, generating text based on input prompts and internal knowledge.

The unique architecture sets RAG-LLMs apart, making them a powerful tool in data analysis.

Overcoming Limitations of Conventional LLMsA common issue with conventional LLMs is their limited access to data, limited knowledge representation, and hallucinations, which result in unrelated or irrelevant responses. By integrating external knowledge, RAG-enabled LLMs overcome these difficulties and give the model access to more recent information beyond its pre-training data, allowing for more coherent, relevant, and factually accurate outputs.

Role of Vector Databases in RAG SystemsVector databases play a crucial role in RAG systems by facilitating efficient retrieval of relevant information. These databases organize textual information into high-dimensional vector representations, enabling fast and accurate similarity searches. Vector representations capture semantic relationships between documents, allowing RAG models to retrieve contextually relevant information for a given query prompt.

Applications of RAG-Enabled LLMs in Data AnalysisYou’ll see how to integrate external data with OpenAI’s LLMs to implement RAG-enabled data analysis in practice. You’ll work with structured (JSON file containing countries’ GDP for 2023), semi-structured (HTML containing info about a country’s economy), and unstructured (PDF containing info about a country’s economy) data.

In your project directory, create the following files and directories:

.├── data│ ├── Economy_of_China.html│ ├── Economy_of_Nigeria.pdf│ ├── gdp_2023.json└── .env└── main.py1 directory, 5 files The main.py file is the project’s entry point, the data folder holds project-related data and the .env file holds the API Key (OPENAI_API_KEY=<key>).

Working With Structured DataStructured data comes in a predefined manner and adheres to a specific schema, making it easy to analyze. Here, you’ll work with JSON data loaded into a pandas dataframe

Add the following code to the main.py file.

```

import the required libraries (1) import osimport pandas as pdfrom llama_index.query_engine import PandasQueryEnginefrom llama_index.tools import QueryEngineTool, ToolMetadatafrom llama_index.agent import ReActAgentfrom llama_index.llms import OpenAIfrom dotenv import load_dotenv## load .env file (2)load_dotenv()## construct file path (3)gdp_file_path = os.path.join("data", "gdp_2023.json")## load data (4)gdp_df = df = pd.read_json( gdp_file_path, orient="records", dtype={"gdp": int}, convert_dates=["year"],)## create query engine (5)gdp_query_engine = PandasQueryEngine(df=gdp_df, verbose=True)## list of query engine tools (6)tools = [ QueryEngineTool( query_engine=gdp_query_engine, metadata=ToolMetadata( name="countries_gdp_ata", description="This provides data regarding the GDP of nations.", ), ),]

``` The code sets up a query engine for the GDP data loaded into a pandas dataframe and defines a tool for querying the data.

Here, we use the PandasQueryEngine as a wrapper interface for the gdp_df dataset to make a query engine (gdp_query_engine). This way, we have an interface that allows asking questions about GDP data using natural language. Using the right query engine, you can create a generic interface for any data you want to interact with.

Using a Python list (tools), we define a list of QueryEngineTool instances, which represent tools for querying the knowledge graph of the external data. QueryEngineTool wraps the gdp_query_engine with metadata. You can have more than one query engine tool. The agent accesses and picks the right tool for a task.

Add the following code to the main.py file below the existing code.

```

load llm and create agent (7)llm = OpenAI(model="gpt-4")agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)## take input prompt (8)while (prompt := input("Enter a prompt (q to quit): ")) != "q": result = agent.query(prompt) print(result)

``` The code creates an agent with a GPT-4 language model and enters a loop that prompts the user for input.

You can use any Large Language Model of your choice by importing the appropriate model class and passing the model’s name.

Using the ReActAgent class, you create an agent to access and interact with the list of query engine tools. The from_tools method initializes the agent with a list of tools and the LLM instance.

Execute the main.py file. As a prompt, input “Which region has the highest GDP?”.

Note how the agent uses the tool we provided in the code to attempt to answer the question asked.

Also, note how the agent takes the necessary steps to answer the question correctly.

Using a Prompt TemplateYou can use a prompt template to provide additional information on processing prompts and queries, provide context, specify the task, guide the response, reduce ambiguity, and improve accuracy. If one is not provided, the default prompt template is used.

Create a prompts.py file in your project directory with the following code.

from llama_index import PromptTemplateinstruction_str = """\ 1. Convert the query to executable Python code using Pandas. 2. The final line of code should be a Python expression that can be called with the `eval()` function. 3. The code should represent a solution to the query. 4. PRINT ONLY THE EXPRESSION. 5. Do not quote the expression."""new_prompt = PromptTemplate( """\ You are working with a pandas dataframe in Python. The name of the dataframe is `gdp_df`. This is the result of `print(gdp_df.head())`: {gdp_df_str} Follow these instructions: {instruction_str} Query: {query_str} Expression: """) You can embed instructions, prompts, and queries within a prompt template, optimizing the input to the agent and response from the agent.

Update the initialization of gdp_query_engine in the main.py file to include an instruction_str:

from prompts import new_prompt, instruction_str......## create query engine (5)gdp_query_engine = PandasQueryEngine( df=gdp_df, verbose=True, instruction_str=instruction_str)gdp_query_engine.update_prompts({"pandas_prompt": new_prompt}) This way, you’ll have provided additional instructions and updated the prompts template.

Execute the main.py file. As a prompt, input “Which country has the highest GDP and which region does it belong?”.

With this, you can customize your prompt templates to fit your data analysis needs and requirements.

Working With Semi-Structured and Unstructured DataSemi-structured data is information that doesn’t necessarily reside in a database but still has some organizational properties that make it easier to analyze. Unstructured data presents a significant challenge in data analysis due to its lack of a predefined data model.

Here, you’ll see how to build an agent to analyze semi-structured data (HTML containing information about a country’s economy) and unstructured data (PDF containing information about a country’s economy) using natural language processing (NLP) techniques, embedding, vectorization, and indexing.

Create a readers.py in your project directory with the following code.

import osfrom llama_index import StorageContext, VectorStoreIndex, load_index_from_storagefrom llama_index.readers import PDFReader, HTMLTagReaderload_dotenv()def create_and_persist_index(data, index_name): print("generating index...", index_name) index = VectorStoreIndex.from_documents(data, show_progress=True) index.storage_context.persist(persist_dir=index_name) return indexdef load_existing_index(index_name): storage_context = StorageContext.from_defaults(persist_dir=index_name) return load_index_from_storage(storage_context)def get_index(data, index_name): if not os.path.exists(index_name): return create_and_persist_index(data, index_name) else: return load_existing_index(index_name)html_path = os.path.join("data", "Economy_of_China.html")china_html = HTMLTagReader(tag="main").load_data(file=html_path)china_index = get_index(china_html, "china")china_engine = china_index.as_query_engine()pdf_path = os.path.join("data", "Economy_of_Nigeria.pdf")nigeria_pdf = PDFReader().load_data(file=pdf_path)nigeria_index = get_index(nigeria_pdf, "nigeria")nigeria_engine = nigeria_index.as_query_engine() The code creates and manages a vector-based store and search index for the HTML and PDF documents. A vector-based store index is a type of data structure and retrieval system that uses vector representations of data to perform searches. This method relies on embedding data into high-dimensional vectors of similar dimensionality that capture the semantic essence of the content, allowing for more subtle and refined searching.

• create_and_persist_index creates a new index from the provided documents using VectorStoreIndex.from_documents, persists the created index to the specified directory (persist_dir=index_name) and returns the index.
• load_existing_index makes a storage context object (storage_context) from the specified directory (persist_dir=index_name) using StorageContext.from_defaults, loads the index from the storage context using load_index_from_storage and returns it.
• get_index checks if an index already exists by looking for a directory with the name index_name. It loads and returns an index if it exists or creates and persists one if it doesn’t.
• As a final step, the documents are being processed. The file paths to the documents are defined. The documents are then read using the appropriate readers (HTMLTagReader and PDFReader). Llama Hub provides different kinds of readers to suit your needs. get_index is called to either create a new index or load an existing one. Lastly, the index is converted to a query engine.

The list of query engine tools in main.py needs to be updated to contain the china_engine and nigerian_engine query engines.

Add the following code to the main.py file.

from readers import china_engine, nigeria_engine......tools = [ ... ... QueryEngineTool( query_engine=nigeria_engine, metadata=ToolMetadata( name="nigeria_economy_data", description="this gives information about the economy of the Federal Republic of Nigeria", ), ), QueryEngineTool( query_engine=china_engine, metadata=ToolMetadata( name="china_economy_data", description="this gives information about the economy of the People's Republic of China", ), ),] Execute the main.py file. It takes a few minutes to generate the embedding, knowledge graph, and index for the documents and store them in the specified folders. This would not be required in subsequent runs except if deleted.

As a prompt, input “What is the difference between the economy of China and Nigeria?”.

As seen, the agent makes use of the tools provided to guide its thoughts and answer the question.

Enhancing the Capabilities of the AgentLlamaIndex provides a FunctionTool class that allows you to wrap a Python function and turn it into a reusable tool or engine.

Let’s implement utility functions that download images from the web and take notes for the user.

Create a utils.py in your project directory with the following code.

import osimport requestsfrom llama_index.tools import FunctionTooldef download_image(url: str): file_name = url.split("/")[-1] response = requests.get(url) response.raise_for_status() file_path = os.path.join("data", file_name) with open(file_path, "wb") as file: file.write(response.content) return file_pathdef save_note_to_file(note: str): note_file_path = os.path.join("data", "notes.txt") if not os.path.isfile(note_file_path): with open(note_file_path, "w"): pass with open(note_file_path, "a") as file: file.write(note + "\n") return notedownlod_engine = FunctionTool.from_defaults( fn=download_image, name="download_image", description="This tool allows the user to download images at a specified URL",)note_engine = FunctionTool.from_defaults( fn=save_note_to_file, name="make_note", description="This tool allows the user to save a text note to a file.",) The list of query engine tools in main.py needs to be updated to contain the downlod_engine and note_engine tools.

from utils import downlod_engine, note_engine......tools = [ downlod_engine, note_engine, ... ... ...] Execute the main.py with the prompts shown in the images below.

A notes.txt file containing the agent’s notes is created in the data folder.

Similar to taking notes, the agent can download an image at a given URL.

You should see an image file, WxHixX4.jpg, created in the data folder.

This way, you can create utility functions (tools) that can perform various tasks, such as data processing, calculations, or interactions with external systems.

ConclusionHere, we’ve covered the concept of RAG-enabled LLMs for data analysis. RAG-enabled LLMs mark a new frontier in leveraging the power of language models for data-driven tasks. By blending LLMs’ natural language processing skills with retrieval methods, RAG models provide a new way to automate data analysis. This combination enables companies to manage datasets of different structures more effectively, uncover valuable insights with less human involvement, and make data analysis available to users of varying skill levels.

The post Using RAG-Enabled LLMs to Automate Data Analysis appeared first on Semaphore.

View Details

Imagine coding side by side with a tireless teammate, who whispers brilliant Ideas, and suggestions in your ear. That’s the luxury GitHub Copilot provides. GitHub Copilot has provided us with relevant code suggestions, and developer assistance, with other helpful features like suggesting code snippets, comments, and documentation based on patterns found in open-source repositories, all while mimicking your coding style.

But here’s the issue: Copilot comes with a monthly subscription fee. If budget is a major concern and you still want to keep up with such a high standard of service, there are free alternatives to consider. One strong contender that comes to mind is Codeium, which offers many of the same code completion and suggestion features as Copilot, with no cost at all.

GoalsIn this article, we will:

• Explore Codeium in detail.
• Provide a step-by-step guide for successful installation.
• Explain how to set up Semaphore with Codeium.
• Highlight other key functionalities and uses.
• Compare Codeium free indivdual plan with GitHub Copilot paid individual plan.
• Discuss how best to use Codeium.

What is Codeium?Codeium is a code completion advanced AI tool, that offers out-of-the-box free toolkits for personal, team, and company usability. With its advancement, Codeium accelerates coding processes, boasting of completing codes in over 70 languages.

Features of CodeiumWe could debate about AI taking our jobs all day, but there is a strong case for its huge added advantage to our productivity as developers, and Codeium hasn’t fallen short of these qualities. With Codeium you have access to;

• Limitless single and multi-line code completions indefinitely.
• Support for over 70 programming languages.
• Seamlessly integrate into all leading IDEs and web editors, including VS-code, Colab, and Jupyter Notebooks.
• Receive assistance via Discord Community.
• Ultra-fast response
• Chatbot for AI suggestions
• Fix code errors, within our IDE
• End-to-end data encryption

Differences Between Codeium and Github CopilotCodeium draws a lot of similarities from Github, but it has its improved uniqueness. We will be comparing Github Copilot’s individual paid plan to Codeiums free individual plan;

PriceCopilot operates on a subscription-based model, priced at approximately $10 per month or $100 per year. Codeium’s Individual plan is entirely free and remains so indefinitely, offering users access to its full range of features without any cost.

FunctionalityCopilot functionality rallies around Single multi-line codegen and chat, on the other hand, Codeium shares the same thing with an added feature of In-IDE integrated search. This integration helps the coding process by allowing developers to have a humanlike convo with the AI or search for additional resources without switching between multiple applications.

Supported IDESCopilot has support for the following IDEs; VSCode, JetBrains, Vim/Neovim, Visual Studio, and Godellum. While Codeium supports; VSCode, JetBrains, Visual Studio, Jupyter/Colab/Deepnote/, Databricks Notebooks, Chrome, Vim/Neovim, Emacs, Eclipse, Sublime Text, and VSCode Web IDEs (ex. Gitpod).

Support for LanguagesGitHub Copilot and Codeium both offer built-in support for popular languages like Python, Javascript, TypeScript, Ruby, Go, Java, PHP, C++, C, Rust, and Swift. However, Codeium provides an added advantage with its large support for emerging or lesser-known languages of over 70+ of these languages.

DeploymentCodeium can be used online, installed on your servers, or within a private network. They’ve passed a security check called SOC 2 Type 2 for their online version. On the other hand, GitHub Copilot has not put out a statement on its security check yet. This could mean that Codeium has been proven to be secure when used online.

InstallationTo have a developer experience we must first install this software. In terms of installing this application, there are usually two ways we can install codeium;

• Playground
• Get Extension

PlaygroundThe first method we will explore will be “trying this out in our browser”:

With this, you can go on a the playground and try codeium in your browser.

Enough of the talks, let’s see how Codeium works, Javascript will be a better choice for me, cause that is what I prefer, you could use another language of preference, and still get the same result. I made edits to the playground, and made a comment that reads “Write a quick todo app in Js”:

Then I went ahead to the next line, and immediately suggestions were given;

You will only need to click tabs to accept suggestions:

Voila! We have our variables to get started. The usefulness of the playground is to convince you to install it in your IDE, and that takes us to the second form of installation

Get ExtensionTo get the extension running in your code editor click on the “get extension” as highlighted below:

The tab below will be open, choose your preferred code editor, there about over 20 code editors available, in this case, we will be using Visual Code Editor for illustration purposes.

When you click your code editor you will be asked to sign up, after signing up you will be able to see the display below, click on a quick install, and it will take you to Visual Studio Code extensions section, follow all instructions and you will have it successfully installed. If you successfully had Codeium’s extension installed your VS code should look like this:

If you missed out on anything during the installation, when you click on the Codeium icon at the left side of your VS code, you will get an error, this is because you may have missed out on linking Codeium to your IDE properly, to fix that, navigate to your Codeium profile, the token will be provided for you there.

To finish up, copy the token, and go to VS Code. In VSCode, open the Command Palette (Ctrl/Cmd + Shift + P), type Codeium: Provide Authentication Token, and hit Enter. This will initiate the authentication process.

Exploring Codeium’s BenefitsThere are various benefits that come with codeium, these benefits are;

Limitless single and multi-line code completions.Codeium can suggest an entire line or even multi-line code blocks. This means Codeium can potentially complete logic structures, or functions without any limitations on the amount of code it proposes.

Imagine you’re building a music recommendation application. Codeium could suggest the entire array of songs, artists, genres, and moods, including the initialization, condition, and body, all at once. Let’s put this to a test:

Single Line:

Multiple Line:

This is the complete code it generated for that function:

Codeium ChatbotCodeium chat can be used to generate boilerplates, explain Code, refactoring Code, and editing code. One fun thing you need to know is that the Chatbot is also called Codeium university, and this is because you can go on an on having a very interactive learning session with this AI.

Generate Boilerplates:

You can easily prompt Codeium to Generate a HTML boilerplate for example and it does this for you. Feel free to copy or Insert directly into your code:

Explanation, Refracturing, and Editing:

To leverage this chatbot feature you can easily select the code block you want to understand or improve. You can highlight any code by dragging your mouse over it, and then right-clicking on the highlighted code block.

This will reveal a context menu with various options, as seen below go down and select either Codeium: refactor selected code or Codeium: explain selected code.

To enjoy Codeium edits, you will also need to highlight the code and then press ctrl + I, this will open a tab like the one below, feel free to prompt as you wish.

Intelligent Search:

Codeium Search helps you find stuff in your code! It currently works with Python, JavaScript, TypeScript, and Go. Here’s how to get started: Go to the Codeium chat prompt it to search your code with natural language, explaining what you want and it will do so!

Downside: Indexing large amounts of code files can take time and slow down your computer, it happens once so no need to fear.

Unit Test and API Integration Made EasyCodeium has an extensive knowledge of various APIs and can seamlessly insert them into your code, right from your IDE. It doesn’t stop there, it can as well generate unit test for your code.

Help developers LearnUsing Codeium you could cut your learning curve so fast. Let’s say for instance you are stuck on a concept or function. Codeium can explain code snippets and logic structures clearly.

Also, the Contextual Code Completion helps with both basic and complex completions. This can expose you to new coding patterns and approaches in such a short time, mind you since it’s trained on a lot of data it teaches you a widely accepted pattern and good code practices.

Product reviewSetting up Semaphore with CodeiumTo set up a Semaphore CI build for a web application with Codeium, let’s get a semaphore account. I prefer using GitHub for my authentications, I will advise you to do so. For this exercise, we will consider a simple example of a Node.js web application using Express.js as the framework that renders Hello, Semaphore, Our build works Now! On the browser(LocalHost).

We can go ahead to create a folder named “semaphorebuild”, doesn’t need to be the best of names. Navigate into the project, Install Express.js, and create an App.js file in the root of our project. Here’s a basic file structure:

Project file structure;

-/semaphorebuild - app.js - package-lock.json - package.json In the app.js file, copy and paste the code below:

const express = require('express');const app = express();app.get('/', (req, res) => { res.send('Hello, Semaphore, Our build works Now!');});const PORT = process.env.PORT || 3000;app.listen(PORT, () => { console.log(`Server is running on port ${PORT}`);}); You can run this using node app.js. In the root of our project, we will create a .semaphore folder, and in it, we will create semaphore.yml file. This file contains the instructions for Semaphore on how to build, test, and deploy our application. We will be prompting Codeium to help with this. Here is the prompt:

Codeuim has given us a template, we can now insert it in our semaphore.yml file:

In our .yml file we need to make a few changes, and here is why; The code template above will only spin up a VM and echo a string “Hello, Semaphore”.

We could take a step further to build it, start the App.js env, and test with a curl cmd that it’s actually up & running.

We want to add three key changes;

• Node.js Installation: This script uses sem-version node to switch to a specific Node.js version.
• Caching: We will use Semaphore’s caching mechanism to speed up the installation of node modules.
• Testing: After starting the server, we will need to perform a curl request to the localHost:3000 route, to be sure it’s up and running.

Feel free to replace the code in the .yml file with this code.

Going further, we will create a repo in GitHub and push the code there. To test our build, navigate to Semaphore Home page.

Create a new project, and follow the provided steps to get your newly created repo linked to Semaphore.

Choose your repo:

Semaphore automatically initializes your project, and check if you have a semaphore.yml file:

Select the “I will use the existing configuration” option above, as Codeium has provided us with that already. You will have to trigger a build by a more recent push to the repository, afterwards, Semaphore builds and gives its results really fast:

With this basic example, we have been able to see how Codeium can help in a real-world scenario, here is the GitHub repo.

Frequently Asked QuestionsWhen thinking about what Codeium can do and why it might be helpful, it’s crucial to realize that it could be used incorrectly or not fully understood. New users might have questions about how to use it properly.

Here are some frequently asked questions to consider, along with answers, to emphasize the importance of using Codeium effectively.

How should developers approach using Codeium, considering it’s an AI?Codeium is an AI tool and should be used with a certain mindset, it’s important to note that Codeium is not a replacement for your coding skills and the critical logic that comes with it.

Use it effectively to boost your productivity and efficiency, but always maintain a focused eye on its suggestions and ensure they align with your coding best practices. While Codeium’s code suggestions are mostly correct, it’s important to remember that it’s an AI tool and may sometimes generate incorrect, repetitive, or instinctive code.

How does Codeium use artificial intelligence to assist with coding?From close experience, Codeium uses AI in two different approaches to become a good coding friend:

• Code Analysis: Codeium utilizes pre-trained AI models that have been fed lots of public code. This training allows them to understand programming language structures, common coding patterns developers use, and even best practices.
• Real-time AI Assistance: Whenever you code, Codeium puts this knowledge to work, by analyzing what you’ve typed and the context of your code and then comparing them with pre-known information that it has been fed already, with this level of knowledge it can then provide intelligent suggestions in real-time.

Is Codeium safe?The simple answer is Yes it is safe, the explanatory answer is that Codeium claims to be SOC 2 Type II compliant, which indicates they have security controls in place to protect user data.

How is Codeium freeThe straightforward answer is, as an individual it is free, but as a team, you will have to patronize with your money. The truth is there is nothing of value that is really free, there will always be a catch, and in this case that catch is, as a team, there is so much you will want to do with software like this, one of its very importance will only be utilized as an organization. This takes us to our next section talking about it’s pricing.

Codeium PricingCodeium has three prices which are the Individual: Free forever, Teams which starts from $12/month and there is Enterprise plan, but the pricing is not available on the website. You can contact Codeium for more information.

Here’s a summary of the features included in each plan:

Individual

• Rapid code autocomplete
• In-editor AI chat assistant
• Repo natural language search
• Unlimited usage
• Trained on open-source data
• Support via Discord.
• End-to-end data encryption.

Teams (all features of Individual)

• Admin usage dashboard
• Seat management and invites
• Upto 200 seats
• Advanced personalization on your codebase.
• Doc searching (coming soon)

Enterprise (all features of Teams)

• Priority support via Slack channel with Codeium team
• Enterprise support portal with documentation and ticketing
• Saas, on-prem air-gapped, and self-hosted VPC deployments available (optional)
• Optional finetuning on your codebase.

ConclusionCodeium offers a promising alternative to Git Copilot, providing developers with an efficient tool for improving coding productivity. While there are other options available, Codeium stands out for its integration with Visual Studio Code and its commitment to data security, Thank you for reading this far, your time was highly appreciated. Please take a look at the Docs as it has extensive use-cases for every IDE.

The post Codeium: The Best Github Copilot Alternative appeared first on Semaphore.

View Details

Have you ever noticed that some websites load really quickly while others take a long time? Caching is a key part of how fast a website loads, and Next.js has powerful built-in features to help you use it well. You can make your app run faster and give users a better experience by optimizing the cache, even if you’re not launching on Vercel while potentially saving on hosting costs. Caching basically saves data that is used a lot so that it can be quickly retrieved. The server can serve a cached version of a page from your Next.js app when a user wants it instead of having to render it from scratch. This translates to:

• Lightning-fast load times: Pages that have been cached load almost instantly, giving your guests a smooth and responsive experience.
• Less work for the server: When you serve stored pages, your server has more resources to do other things, which makes your application more scalable

To put it simply, optimizing the cache keeps your Next.js app going at its fastest, which makes users happier and lowers server costs.

Next.js gives you a number of ways to improve your cache in a setting that doesn’t depend on Vercel. It automatically caches statically generated pages, improving performance for frequently accessed parts of your application. You can also control data fetching cache behavior by specifying the duration the browser stores the data, ensuring it stays fresh while optimizing speed. For advanced scenarios, you can build custom caching logic using libraries or techniques like local storage. By utilizing Next.js’ built-in caching mechanisms and data fetching strategies, you can greatly enhance your application’s performance without depending on an external platform like Vercel.

Static Site Generation (SSG)The practice of assembling and displaying web pages at build time as opposed to on-demand is known as static site generating. With HTML, JavaScript, and CSS among other static files, this produces a set that is prepared for user delivery. SSG is accomplished in Next.js via use of functions like getStaticProps and getStaticPaths. Run during build time, these routines retrieve data and produce the HTML required for every page. You can retrieve data and send it as props to your page with getStaticProps, so the content is ready even before the user requests it.

Benefits of Build Time Caching of Static HTMLPre-rendering of your information gives SSG a number of benefits:

• Lightning-Quick Load Times: Your visitors will have a seamless experience as pages load nearly instantaneously because the HTML is already built.
• Greater Scalability: Because SSG doesn’t have to make content for each request, it lightens the strain on your server. This makes traffic handling by your application more efficient.
• Improved Search Engine Optimization: Pre-rendered content allows search engines to crawl and index your website faster. SSG may result in lower server costs because of a lesser server load.

Implementation Tips and Best Practices for SSG in Next.jsNow that you understand the core SSG process, here are some key implementation tips to optimize your workflow:

• Choose the Right Content: Periodically updated material, such product pages or blog posts, is best served by SSG. It works less well with often changing material.
• Use getStaticProps: This Next.js function is your workhorse during the construction process. Use it to retrieve data and make it available to your page components. Access the source code via github

export async function getStaticProps(context) { // Fetch data here const res = await fetch('https://api.example.com/data'); const data = await res.json(); // Return as props return { props: { data }, };} * Dynamic Routes: For dynamic pages, use getStaticPaths and indicate which routes need to be pre-rendered:

export async function getStaticPaths() { const response = await fetch('https://api.example.com/products'); const products = await response.json(); // Set fallback: false to indicate all pre-defined paths are generated at build time return { paths: paths, // Assuming 'paths' is defined elsewhere with product IDs or slugs fallback: false, }; } Incremental Static Regeneration (ISR)ISR builds on the foundation of SSG. During the build process, Next.js renders your application’s HTML pages with the most recent data. These pre-built pages are subsequently served to users, providing extraordinary speed. While ISR uses pre-rendered pages in the same way that SSG does, it adds dynamism. When a request for a page is received, Next.js examines the cache. If the cached page is fresh (within a defined time frame), it is served immediately. If the cached material is obsolete, Next.js initiates a background re-generation process to retrieve new data and update the HTML content. The user continues to see the cached page while the update occurs in the background, ensuring a smooth experience.

It’s important to understand that ISR doesn’t achieve real-time updates on static pages at runtime. Why? This is because while the page is being regenerated in the background, the user continues to see the cached version. This re-generation can take a few seconds, depending on how you retrieve data. The revalidation window you set determines how fresh the item is. By default, there is no revalidation, but you can choose a time period (for example, 60 seconds) for Next.js to check for updates. This means that the content may be slightly out of date compared to a really real-time system.

How to implement ISR in Next.js applicationsNext.js makes implementing ISR straightforward. Here’s a basic example:

export async function getStaticProps() { const response = await fetch('https://api.example.com/data'); const data = await response.json(); // Revalidate the data every 60 seconds (can be adjusted) return { props: { data: data, }, revalidate: 60, // In seconds };} In this example, getStaticProps retrieves and returns data as props. The revalidate parameter controls how frequently Next.js checks for updates in the background. When new data becomes available, the cached page is automatically refreshed. Incremental Static Regeneration allows you to build Next.js applications that are both fast and keep your content fresh. Understanding ISR’s strengths and limits allows you to properly use it to produce a seamless and up-to-date user experience.

Cache-Control HeadersCache-Control headers are necessary for managing the caching behavior of online resources. They teach browsers and intermediary caches such as CDNs how to handle caching, which can have a substantial impact on a website’s performance and efficiency. They determine how, where, and how long a resource should be cached. They help to reduce bandwidth utilization and server burden, resulting in faster load times. They ensure that consumers receive the most current material without excessive server requests.

Setting Up Cache-Control Headers in Next.jsWhile Next.js automatically configures some default Cache-Control headers, you may want to change them for specific cases. Here is a summary of the options:

• During SSG, you can define Cache-Control headers within the getStaticProps function. This approach is ideal for static content with a set expiration time.

export async function getStaticProps() { // ... fetch data return { props: { data: data, }, revalidate: 60, // Optional: revalidate every 60 seconds with ISR cacheControl: 'public, max-age=3600', // Cache for 1 hour };} * For API routes, you can set headers directly in the response object.

export default async function handler(req, res) { const data = await fetch('https://api.example.com/data'); res.status(200).json(data).setHeader('Cache-Control', 'public, max-age=86400'); // Cache for 1 day } Best Practices for Cache-Control Headers* Differentiate between static content (e.g., blog articles) and dynamic content (e.g., user profiles) and use appropriate headers. Static material can have longer cache periods, whereas dynamic content may need shorter caching or none at all. * Set appropriate max-age values based on your content’s update frequency as it specifies how long a resource can be cached before being considered stale * Use the immutable directive for really static items that never change (for example, photos or JavaScript files). This informs caches to never re-validate the resource, optimizing performance.

export async function getStaticProps() { // ... fetch data (for non-immutable content) return { props: { data: data, }, cacheControl: 'public, immutable', // Never revalidate }; } * Use no-cache and no-store together with appropriate revalidation mechanisms (for example, SSR or ISR) to ensure that users always view the most recent version. However, use caution because they can have a considerable impact on performance if used excessively.

export async function getServerSideProps() { // ... fetch data (for dynamic content) return { props: { data: data, }, cacheControl: { // Revalidate every 60 seconds with ISR (consider appropriate strategy) maxAge: 60, noCache: true, // Invalidate cached data but doesn't prevent caching entirely noStore: true, // Don't store data in any cache (use with caution) }, }; } * Try out several Cache-Control header settings and monitor real-world performance from browser developer tools in the network and timing tabs or tools like Google PageSpeed Insights, Lighthouse and so on to determine the best mix between the benefits of caching and the freshness of the content.

Client-Side CachingClient-side caching includes storing data obtained from the server on the user’s browser. This cached data can then be utilized for future requests, avoiding the need to retrieve it from the server. Typical client-side caching methods include:

• Browser Cache (HTTP Cache): Using Cache-Control headers from the server, the browser stores resources in cache on its own. Some of these headers are already handled by Next.js by default.
• Scripts known as “service workers” operate in the background of a webpage even when it isn’t being used actively. If there is cached data available, they can serve it and intercept network queries.
• Web Storage: There are two primary choices for web storage offered by the browser:
  • localStorage: Information is kept around even when the browser window is closed.
  • sessionStorage: When a browser window or tab is closed, data is removed.

Implementing Client-Side Caching in Next.jsHere are two approaches to implement client-side caching in your Next.js application:

1. Service Workers for Granular Caching: Service workers provide you more control over how you cache. To add service worker to your Next.js file, we must first register service worker in a global scope environment and create a service worker file that contains all of our service worker code. You can access the code through GitHubFirst, let’s register a service worker in _app.js by adding the following in the function:

useEffect(() => { if ('serviceWorker' in navigator) { navigator.serviceWorker .register('/service-worker.js') .then((registration) => console.log('scope is: ', registration.scope)); } }, []); We are registering the service worker in _app.js to enable application-wide caching. This ensures that the service worker has access to all resources (pages and assets) in your Next.js project. This allows it to possibly cache data and requests for whatever page the visitor visits. Head over to the public directory and create a new file called service-worker.js. This file serves as the control center for our client-side caching implementation. Paste the following in the file you created

const installEvent = () => { self.addEventListener('install', () => { console.log('service worker installed'); });};installEvent();const activateEvent = () => { self.addEventListener('activate', () => { console.log('service worker activated'); });};activateEvent(); After saving service-worker.js, reload the browser page with “Offline” unchecked in DevTools. Enable “Update on reload” to ensure the service worker updates with each page refresh. The status below shows that the service worker is activated and running.

To accomplish efficient client-side caching for the current user’s page, we use the clone() method from the service worker’s fetch event handler. This allows us to cache the entire page’s assets at once. Add the following to your service-worker.jsfile

const cacheName = 'v1';const cacheClone = async (e) => { const res = await fetch(e.request); const resClone = res.clone(); const cache = await caches.open(cacheName); await cache.put(e.request, resClone); return res;};const fetchEvent = () => { self.addEventListener('fetch', (e) => { e.respondWith( cacheClone(e) .catch(() => caches.match(e.request)) .then((res) => res) ); });};fetchEvent(); We will have the following in your cache and you can now select the “offline option” and reload the page.

1. Local Storage for Smaller Cached Data in Next.js: You may store key-value pairs of data straight inside the user’s browser using the web storage API localStorage. In contrast to session storage, which disappears when the browser window or tab is closed, this data remains when the browser window is closed. It saves user preferences like as language settings, themes, and accessibility options to ensure a tailored experience across sessions. Temporarily save partially completed forms to avoid users from losing information if they mistakenly navigate away. To keep user login state across page refreshes, it securely store authentication tokens (encrypted properly).Here’s a basic example demonstrating how to use localStorage to cache data in a Next.js component. We will create a custom hook to access the local storage with the following code. You can access the code via Github.

import { useState } from "react";const useLocalStorage = (key, initialValue) => { const [state, setState] = useState(() => { // Initialize the state try { const value = window.localStorage.getItem(key); // Check if the local storage already has any values, // otherwise initialize it with the passed initialValue return value ? JSON.parse(value) : initialValue; } catch (error) { console.log(error); } }); const setValue = (value) => { try { // If the passed value is a callback function, // then call it with the existing state. const valueToStore = value instanceof Function ? value(state) : value; window.localStorage.setItem(key, JSON.stringify(valueToStore)); setState(value); } catch (error) { console.log(error); } }; return [state, setValue];};export default useLocalStorage; In index.js, you can use it as follows:

import useLocalStorage from "@/hooks/useLocalStorage";import { useState } from "react";export default function Home() { // Get the value from local storage if it exists const [value, setValue] = useLocalStorage("favoriteNumber", ""); // Set the favorite number state using the value from local storage (or the empty string) const [favoriteNumber, setFavoriteNumber] = useState(value); // When user submits the form, save the favorite number to the local storage const saveToLocalStorage = (e) => { e.preventDefault(); setValue(favoriteNumber); }; return ( <div> <label htmlFor="number">Your favorite number</label> <form onSubmit={saveToLocalStorage}> <input id="number" value={favoriteNumber} onChange={(e) => setFavoriteNumber(e.target.value)} /> <input type="submit" value="Save" /> </form> </div> );} CDN IntegrationPicture a network of servers positioned thoughtfully throughout the globe. Such is a Content Delivery Network (CDN). By integrating a CDN with your Next.js application, you essentially spread your static content—images, JavaScript, and CSS—over these servers. By bringing your material closer to users, loading times are greatly accelerated.

Benefits of CDN Integration Faster Load Times: By serving material from the closest CDN server, users experience is delivered with less latency. * Greater Scalability: High traffic spikes can be managed by CDNs without affecting your primary server. * Reduced Bandwidth Costs: Your server will be less taxed when static material is offloaded to the CDN, which may result in lower bandwidth bills. * Increased Availability:* Because CDNs are dispersed geographically, your material is still available even in the event that one server goes down.

How to Integrate a CDN in Next.js ApplicationsNext.js provides a simple method for CDN connection. This is the broad outline:

1. Select and configure your CDN provider. Examples of CDN providers are Cloudfare, Fastly all of which offer significant benefit over Vercel. You’ll be given a CDN URL to serve your assets from.
2. Within your next.config.js file, specify the assetPrefix. Next.js is instructed to add this URL to the paths of your static assets. An illustration setup is this:

module.exports = { // ... other configurations assetPrefix: 'https://your-cdn-domain.com/',}; 3. Replace ‘https://your-cdn-domain.com‘ with the URL provided by your CDN and deploy your application to a platform that supports CDN integration. You can get more helpful details here.

ConclusionCache optimization in Next.js is a complex technique that greatly improves cost effectiveness as well as performance. Pre-rendering pages at build time with Static Site Generation (SSG) allows developers to guarantee quick delivery and less server load. This method is improved even more by Incremental Static Regeneration (ISR), which permits real-time page updates without compromising the advantages of static generation.

Cache-Control headers must be used strategically in order to control content storage and delivery and to allow developers to customize caching behavior to suit the requirements of their applications. Using Service Workers and localStorage to cache browsers on the client side can dramatically shorten load times and enhance user experience.

Ultimately, connecting with Content Delivery Networks (CDNs) broadens the use of cache optimization by dispersing content over a worldwide network to guarantee quicker access from any area. Next.js’s all-encompassing caching method not only speeds up content delivery but also makes best use of available resources, making it a necessary technique for contemporary web development.

The post Cache Optimization on NextJS Without Vercel appeared first on Semaphore.

View Details

Testing remains a critical yet often misunderstood and undervalued component of software development. What’s more, as artificial intelligence threatens to reshape the role of testers, questions of job security and demonstrating value become increasingly pressing. From the struggle for recognition and early involvement in the development process to the delicate balance between automation and human insight, testers navigate a minefield of technical and organizational hurdles. Michael Larsen, a veteran with over three decades of experience in the field, sheds light on the complex challenges facing the software testing industry today.

Edited transcriptionMichael Larsen’s experience using computers was mostly limited to music production when he started working at Cisco in the early 1990s as an engineer. Soon enough, his aptitude and interest moved him to work as a tester.

After a decade at Cisco, Michael sought to broaden his horizons and transitioned to independent consulting, a path he continues to embrace today, even with the realities of an unstable job market. “After a 30-year career,” Michael jokes, “I tell people that I am an independent software testing consultant because it sounds better than saying that I’m basically unemployed, which is the truth.”

Software Testing Strategies: Skills and ethics for testersTo share his vast experience in the software industry and testing practice, Michael recently co-authored with Matthew Heuser Software Testing Strategies. Published by Packt Publishing, the book delves into various aspects of testing, making it a valuable tool for testers of all levels.

Software Testing Strategies consists of 16 chapters divided into three parts. The first part consists of testing knowledge and skills for testers; “things you should know as a tester,” says Michael. While experienced testers might find this section familiar, it includes a valuable chapter on specialized testing in in-demand areas like modern web applications, SaaS, microservices, and APIs. Michael clarifies that the book does not attempt to turn readers into experts on these technologies. Instead, it provides a solid base for understanding how testing applies to them: “You fired up a tool, you know how to use it superficially. That’s great. There’s a whole lot more you need to learn before you can say that you’re proficient at it.” Hence, he encourages exploration and further learning beyond the book’s introductory guidance.

The book’s second part dives into software delivery processes and the testing strategies that work best within those frameworks. Michael stresses the importance of flexibility and adaptation, rejecting a “one-size-fits-all” approach to testing. He uses a recipe analogy to illustrate this point. Testing, like a recipe, has core principles that remain constant. A deep understanding of these core principles allows testers to improvise and test software in various contexts, coming back to them when needed.

Besides, once you have a grip on how this recipe works, you can tweak it to match your requirements: “You can change the flavor profile a little bit or the development environment has changed and what used to work with the team that you were with before may not work here,” Michael affirms. A deep knowledge of what underlines the “recipes” makes it possible to adjust it to suit your space problem. Michael believes it is necessary to understand the core principles of testing (“recipes”) to adapt and improvise in different environments.

The final section of Michael’s book ventures into the often-overlooked territory of the “political landscape” of software testing. Here, he tackles the undervaluing of testers, the ethical dilemmas they face, and strategies for navigating challenging situations. This last section, Michael admits, has garnered the most interesting feedback, as it addresses crucial issues rarely explored: “Let’s talk about the difficult people you’re going to have to deal with, the difficult conversations you’re going to have, the fact that you may be asked to compromise your ethical standards or somebody might term something in a way where you’re doing what you think is the right thing and somebody else is misrepresenting your work and you’re getting thrown under the bus when something goes wrong.”

Empowering testers for quality, usability, and inclusivityFrom his experience in the industry, Michael doesn’t hide that testers can be “very highly valued or can be denigrated, treated like a second class citizen.” Yet Michael defends the tester role as irreplaceable for achieving quality, provided certain conditions are met. “If you’re not willing to give the testers autonomy and they don’t get the chance to make the decisions or they don’t get a chance to be part of early conversations, then no, we can’t really assure quality; we can’t even really provide you with good testing,” says Michael.

Quality hinges on empowering testers with autonomy and authority over decisions, and also, on their involvement in early project discussions. According to Michael, testers are most valuable at the beginning of the development process, “before line one of a feature is even written.” Their role is to ask critical testing questions from the outset, acting as a voice of caution that can ultimately prevent future problems. Testers should be involved in defining how features will be tested, including considerations for APIs and potential user scenarios, and proactively examining “what if” questions, which significantly contribute to a well-rounded product.

The involvement of testers in development isn’t limited to identifying potential problems but also to anticipating a wider range of user experiences. In this regard, Michael challenges the common perception of accessibility features being solely for people with disabilities. While accessibility features like screen readers or closed captions are typically seen as aids for those with visual or hearing impairments, “everybody can benefit from accessibility enhancement because every one of us can find ourselves in a disadvantageous situation and we don’t have to be disabled to be in it,” Michael points out. Even a well-designed, accessible product can become unusable due to external factors. For instance, bright sunlight can make it difficult to see a screen, or a noisy environment can render audio features useless. In these scenarios, accessibility features act as a safety net for every user, ensuring usability even in less-than-ideal conditions.

However, Michael also cautions that a highly compliant product can still fail if it’s simply unpleasant to use: “You can have the most compliant software product…but if it’s terrible to use, if people don’t like interacting with it, then you’ve won the battles but you’ve lost the war.”

Is test automation overhyped?Technology hype cycles end up in two ways, they either don’t stick around or “they work with the way you do things.” In the case of test automation, by itself it has been around for a long time, still, Michael argues it has limitations: ” Automation can do a lot of neat things. It can’t do everything.”

Automation is great for repetitive tasks and following established paths, as it ensures reliable and consistent testing of programmed functionalities. However, once an automated test is written and integrated, the “interesting work” of discovery is mostly done, as automation is incapable of uncovering new issues. Test failures often indicate code changes or library updates, not necessarily new product issues.

Fixing these failures becomes change management, not an active exploration of the product. Hence, Michael understands that at this point automated tests become maintenance tools that check if things work the same after code changes. Besides, in this stage automated tests are also subject to diminishing returns. All they can do is manage known test cases, however, automating every possible test case might not be necessary or efficient. Sooner or later will come a time where the time and effort invested in automation doesn’t yield significant benefits.

In this way, Michael believes it is necessary to implement automated testing specifically where it is more critical. For example to prioritize critical tests that ensure core functionalities work as expected. What’s more, Michael believes these critical tests should be integrated into the development pipeline to catch regressions early. Staging environments can’t perfectly replicate real-world conditions. Production testing helps identify user behavior and the adoption of new features. Besides, users and unexpected situations can reveal issues missed in controlled environments.

The place of AI in testing“If a technology can come along that can do a lot of what they feel you can do and save a lot of money, any logical organization is going to do that,” says Michael on the rise of AI in test case generation. Still, he sees AI as freeing testers from repetitive and manual labor and forward to more challenging issues. Being responsible for more complex issues can be intimidating, however, there’s no alternative: “I’m going to have to think about what value I bring now that say chat GPT or copilot can’t do exactly what I’m doing. If the answer is nothing, you’re in a dangerous position,” says Michael.

Since testers need to showcase their unique value proposition beyond what AI can do, another way to look at complexity in testing, and that can make testers value their work properly in the face of becoming obsolete, is seeing testing as risk management. Regarded as such, testing is not about “going through the motions and hammering on the product”, but, as Michael puts it, about “understanding what are the risks that we face.”

The bottom lineYou can follow Michael on Linkedin. To learn more about his work, you can read his testing blog TESTHEAD and listen to his podcast, The Testing Show.

Grab a copy of his co-authored book, Software Testing Strategies, published by Packt Publishing. Discussion groups about the book are also available on Facebook and LinkedIn.

If you are a hard rock fan, check out Michael Larsen’s band, Ensign Red, on Spotify, Apple Music, and Amazon for a taste of his musical side.

The post Michael Larsen on Quality Assurance in The Age of AI appeared first on Semaphore.

View Details

Bootstrapping a tech startup by yourself or with a small team with no money is no easy feat, but it is not impossible. In this article, I’ll highlight how you can save your cash resources when trying to start your startup and the right mindset you should have.

Being a tech entrepreneur is one of the most exciting and rewarding ventures you can embark on, whether you’re a developer or someone interested in starting a technology business. It’s fascinating and beautiful to see something you started come to fruition as people pay to use your product. But how do you plan a startup when you don’t have any money?

Is the problem real? Will target customers pay for your solution?Is the problem you’re trying to solve real? This question is shockingly important, as many founders create products that don’t solve any real issues, or solve problems that people are not willing to pay to resolve. I recently conducted a survey for a product idea I had, and I was surprised to find that while the participants acknowledged the problem my product aimed to solve, they weren’t willing to pay for the solution.

The leading cause for startup fail is lack of market need. In other words, about 42% of startups fail because they find no customers willing to pay for the proposed solution.

For your startup to make any profit at all, you need to build something that people are willing to spend their money on, if not it will likely fail.

Team and PlanningWhen establishing a team for a startup that does not involve substantial money, the most important issues are strategic planning and resource management. It is wise to have a small and flexible team with varied skills that are important in the first steps of the startup. Many times, this involves having co-founders or first employees who can take on different hats, such as designing and marketing the products, while at the same time handling operations activities.

If you’re not a technical founder, and you don’t want to get bugged down by taxes and organization structure, you can find yourself a technical cofounder at techcofounder. If its someone you’re acquainted with, that’s even better, as it makes communication and decisions much easier.

Since you’re trying to get up and running quickly, it’s best if you get a cofounder that is really technical and know what they are doing and wouldn’t need to get another set of helping hands since they are able to wear many hats in the organization. Another way you can go about this if you have the time is to learn how to code. This route is time-consuming, and it’ll be smart to have another source of income to sustain yourself while you learn to code and build your startup by the side.

On the other hand If you’re a technical founder seeking a non-technical co-founder, the challenge lies in effectively selling your vision. Successfully doing so can significantly reduce startup costs and the time needed to develop your MVP.

Let’s get into fun part of how you can spend little to no money by leveraging some free tools when building your startup.

How to spend little to no money?Many people give up before they even try to build something because of the mindset they’ve created, thinking they need a lot of money to start. Their grand idea is almost immediately dismissed due to concerns about the resources they’ll need to make their dream a reality. In this section, we’ll discuss how to start with almost no money. We’ll cover starting lean and later touch on open-source tools that you can leverage for free in your startup journey.

Build an MVPSpeed is very important when trying to ship your product. Because of this, you’ll need to prioritize building a minimum viable product for your business. A minimum viable product should contain the main features that solve the problem your target customers have. Any other feature that does not solve that problem should be on the sideline. In the words of Eric Ries who wrote The Lean Startup

The lesson of the MVP is that any additional work beyond what was required to start earning is waste, no matter how important it might have seemed at the time.

As a founder it’s easier to build out your MVP if the you have personally faced the problems you are trying to solve. This experience with the problem makes you really experienced in tackling the problem by drafting a quick solution through your MVP.

Adopt an iterative development process, launching with a basic yet functional version of your product, and refine it based on feedback from initial users. This method not only saves upfront costs but also ensures that further investments are guided by actual user demand and feedback, maximizing every dollar spent.

Leverage free and open-source software and toolsWhen building your startup, try your best not to drop a ton of money on expensive tools and services just to get your product live and working, instead utilize free and open-source software tools and services for development, design, and project management to cut costs significantly.

Let’s look at some of these tools below:

1. Developing Your Product: Use open-source software like Git for version control and GitHub for repository hosting. Use frameworks like React, Vue.js, or Angular for front-end development and Node.js, Django, or Ruby on Rails for back-end development.
2. Designing on a Dime: Use Krita for image editing and digital painting. For designing sleek vector art or layouts, try Inkscape. Additionally, if you’re looking for a collaborative UI/UX design tool for your team, a good choice would be Figma.
3. Setting Up Shop Online: Build your website with WordPress. It’s not just for blogs—you can create full-fledged business sites with it, and there’s a huge community for support.
4. Staying Organized: Keep your projects and tasks in line with tools like Trello or Asana. They’re great for mapping out what needs to get done and tracking progress. Notion is another excellent tool for project management and documentation. Basecamp offers an affordable plan, and Operately, which is being developed by the founders of Semaphore, is a promising open-source option.
5. Communication is Key: Don’t let communication costs creep up. Use Discord or Slack for messaging; both are perfect for small teams.

Marketing and scaling at no costIn your company’s early days, you need to get creative and cut costs by leaning into low-cost marketing ideas for startups. Think PR, partnerships, content marketing, and crowdfunding to get your product out there without spending a ton. These strategies are key to ramping up your growth and boosting your revenue without the hefty price tag.

Here are some tips to save cost while trying to market and scale your startup:

• Leverage Social Media: Utilize platforms like Instagram, Twitter, and LinkedIn to share engaging content about your startup. Regular posts, engaging with other users, and using relevant hashtags can help spread the word without any cost.
• Content Marketing: Start a blog or YouTube channel related to your industry. Providing valuable content can attract an audience, build your brand, and establish authority in your field.
• Networking: Attend free industry meetups, webinars, and other events to network with potential customers and partners. Building relationships can lead to opportunities and exposure.
• Partnerships: Collaborate with other businesses to offer joint promotions or barter services. This can open up your products to new audiences without incurring direct marketing costs.
• Customer Referrals: Encourage satisfied customers to refer others. Offering incentives can boost this effort, which can be a powerful tool for organic growth.

Having the right mindsetBefore you start building on your startup, it’s important to have the right mindset. Resilience and a strong problem-solving attitude—as these often determine your success. Many founders falter due to a lack of resilience in following through with their plans. A resilient mindset acts as an antidote to skepticism, powering you through your startup journey.

While building your startup, you may face numerous challenges including failures, setbacks, and rejections. Having a resilient mindset during this period can be crucial to whether your startup fails or succeeds. You also need strong problem-solving skills that focus on actual problems needing resolution. Keep your focus on how your product or service can benefit your customers.

I’ll end the article with this quote by Nelson Mandela

“Do not judge me by my successes, judge me by how many times I fell down and got back up again.”

ConclusionIn this blog post you’ve learned and gotten insight on how to build your startup with little to no money and the right mindset to have when building a tech startup. Hopefully, the knowledge gained from this tutorial would be able to guide you in your journey as a successful entrepreneur.

The post Zero Budget Big Dreams: Starting a Tech Startup from Scratch appeared first on Semaphore.

View Details

Large Language Models (LLMs) are becoming more and more employed in various applications, ranging from natural language processing to predictive analytics.

Considering the resources needed to use LLMs, in this article we’ll discuss how to keep down the costs associated when self-hosting LLMs. In the case of self-hosting, LLM costs, in fact, can be prohibitive, especially for mid-sized companies. So, in this article, we provide practical strategies to help you manage and reduce the costs associated with self-hosting LLMs.

Strategy 1 to Keep Self-hosted LLM Costs Down: Optimize Model SelectionThe first step to keep LLM costs down is selecting the right model for your needs. Not all applications, in fact, require the largest, most powerful models available. So, here are some considerations to keep in mind:

• Performance requirements. Determine the level of performance you need. For example, if your application involves basic text classification or sentiment analysis, smaller models like DistilBERT might be sufficient. For more complex tasks, instead, such as generating coherent text or answering detailed questions, larger models like GPT-3 may be necessary.
• Model size and capabilities. Larger models require more computational resources and this increases the costs. So, evaluate models based on their size, number of parameters, and specific capabilities to ensure you are not overpaying for unnecessary features.
• Domain-specific models. Consider models that are fine-tuned for your specific domain. These models often provide better performance with fewer resources compared to general-purpose models. For example, if you work in the biomedical field you might find useful the BioBERT model, which is a pre-trained biomedical language representation model based on BERT, designed to handle various biomedical text mining tasks such as named entity recognition, relation extraction, and question answering. Or, if you work in the financial industry, you may find interesting the FinBERT model, which is a pre-trained language model specifically designed for financial sentiment analysis, particularly suitable for tasks such as sentiment classification, financial text summarization, and risk assessment.
• Open-Source Alternatives: Explore open-source models that can be hosted on-premises. These models often have lower licensing costs and provide more flexibility in terms of deployment and customization. BERT, among the others, is one famous example of open-source LLM.

Strategy 2 to Keep Self-hosted LLM Costs Down: Efficient Resource AllocationConsidering the fact that LLMs use a huge amount of computational resources, proper resource management is essential for reducing the costs of self-hosting these models, considering the cost of the hardware.

Here are some strategies to optimize hardware usage to save costs:

• Leverage GPUs effectively. To self-host LLMs you need machines that use GPUs, but they are expensive. So, to save costs on GPUs the main idea is to utilize GPU resources effectively by ensuring high utilization rates. This might involve implementing techniques and methodologies like batching requests, optimizing the code to reduce idle times, and using GPU-sharing techniques such as multi-GPU environments.
• Dynamic resource management. Use virtualization to dynamically allocate resources to the LLM based on demand. Tools like VMwareor Proxmox allow you to create virtual machines (VMs) that can be scaled up or down depending on current needs.
• Containerization. As containers can be spun up or down dynamically on your self-hosted infrastructure, you can implement a container orchestration strategy with Kubernetes or Docker Swarm to manage resource allocation efficiently. Containers, in fact, allow for better resource allocation and can be easily moved across different environments, ensuring optimal usage of available resources.
• Implement data parallelism methodologies. Data parallelism involves splitting the data across multiple GPUs or machines, allowing each GPU to process a portion of the data simultaneously. This approach can speed up the training and inference processes, provinding also an efficient use of the available hardware resources. Depending on your hardware setup and the framework you use, to implement data parallelism Pytorch provides the torch.nn.parallel.DistributedDataParallel class while Tensorflow provides the tf.distribute.MirroredStrategy class.

As an example, let’s consider the implementation of data parallelism methodologies with Pytorch.

First of all, you have to discern based on your architecture. The cases are:

• Single-machine with a multi-GPU setup.
• Multi-machine setup.

NOTE: if you haven’t already done so, you need to install Pytorch

Single-machine with a multi-GPU setup: In a single-machine setup with multiple GPUs, you can use frameworks like PyTorch (or TensorFlow) to distribute batches of data across GPUs. Each GPU processes its batch independently, and gradients are aggregated and averaged before updating the model parameters.

The implementation could be something like this:

import torchfrom torch.nn.parallel import DataParallel# Initiate the LLM modelmodel = YourLLM-Model()# Parallelize model across GPUsmodel = DataParallel(model)# Move model to GPUsmodel.to('cuda')# Iterate over data loader for data in dataloader: inputs, labels = data inputs, labels = inputs.to('cuda'), labels.to('cuda') outputs = model(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() The “magic” with the above code happens thanks to:

• model = DataParallel(model): the LLM model is wrapped with the method DataParallel(), parallelizes the model across multiple GPUs. This means that the input data will be split and processed in parallel by different GPUs, speeding up the training process.
• model.to('cuda'): the LLM model is moved to the GPU by calling the .to('cuda') method. This ensures that the model computations are performed on the GPU (which is faster than the CPU for deep learning tasks).
• The iterations over the dataloader: The dataloader is an iterator that provides batches of input data and labels. In each iteration, it yields a new batch.

Multi-machine setup: For multi-machine setups, use torch.distributed in PyTorch or tf.distribute.MultiWorkerMirroredStrategyin TensorFlow. This approach synchronizes data processing across machines, making efficient use of distributed hardware resources.

For example, the implementation could be something like this:

import torchimport torch.distributed as distfrom torch.nn.parallel import DistributedDataParallel as DDP# Initialize the process group def setup(rank, world_size): dist.init_process_group("nccl", rank=rank, world_size=world_size)# Destroy the process group (to clean up resources used for distributed training)def cleanup(): dist.destroy_process_group()# Train the LLM modeldef train(rank, world_size): setup(rank, world_size) model = YourLLM-Model().to(rank) ddp_model = DDP(model, device_ids=[rank]) optimizer = torch.optim.Adam(ddp_model.parameters()) for data in dataloader: inputs, labels = data inputs, labels = inputs.to(rank), labels.to(rank) outputs = ddp_model(inputs) loss = criterion(outputs, labels) optimizer.zero_grad() loss.backward() optimizer.step() cleanup()if __name__ == "__main__": # Define the number of GPUs or nodes world_size = 4 # 4 is just a representation # Handle the parallel execution mp.spawn(train, args=(world_size,), nprocs=world_size, join=True) So, in this case, the code initiates and destroys the processes to save resources and distribute them among different machines. Also, note that, in this case, model = YourLLM-Model().to(rank) moves the model to the appropriate GPU specified by rank, which represents a unique (integer) identifier for each process within a distributed system, that ranges from 0 to “world_size – 1″.

For example, in a machine with 4 GPUs, ranks 0, 1, 2, and 3 would typically map to GPU 0, GPU 1, GPU 2, and GPU 3 respectively.

Strategy 3 to Keep Self-hosted LLM Costs Down: Implementing Model Compression TechniquesAnother way to reduce LLM costs when self-hosting is to use model compression techniques.

Here are some of the most used techniques:

• Quantization. Quantization is a methodology that reduces the precision of the model’s weights from 32-bit floating-point to 16-bit or even 8-bit integers. This decreases the model size and speeds up inference times, leading to lower resource usage. Since, in the case of LLMs, computational models are represented in the memory of GPUs as tensors, to store them we can use different data types like Float64, Float16, or integers. Since the data type you choose impacts the number of digits used in memory, shortening digits passing from floats to integers, for example, reduces memory utilization.
• Pruning. Pruning is a methodology that involves removing less important weights from the model to reduce the model’s size while maintaining performance within acceptable limits. There are different ways to prune LLMs, most of them still at an experimental stage. The LLM-Pruner is one of them; it tackles the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset.
• Knowledge distillation. This technique involves training a smaller model (called the student) to replicate the behavior of a larger model (called the teacher). In other words, This technique involves using a smaller model that can achieve comparable performance with a fraction of the computational resources. The main idea behind knowledge distillation is to transfer advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterparts like LLaMA and Mistral.
• Low-rank factorization. Since LLMs are Neural Networks that contain dense layers performing matrix multiplication, the idea behind low-rank factorization is to decompose the weight matrices of the neural network into products of smaller matrices. This way, we reduce the number of parameters and computations required, leading to faster and more efficient models.
• Sparse representations. Utilizing sparsity with neural networks can reduce the model size and computation time, especially with LLMs. Typical techniques to use can be L1 regularization and using Sparse Transformers.

As an example, let’s consider how to apply the quantization technique to an LLM using Pythorch.

But first, if you haven’t already done so, you need to install the transformers library:

pip install torch transformers Now, let’s see the quantization example:

import torchfrom transformers import DistilBertModel, DistilBertTokenizer# Load the tokenizer and modeltokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')model = DistilBertModel.from_pretrained('distilbert-base-uncased')# Tokenize sample input textinput_text = "Quantization can help reduce model size and improve inference speed."inputs = tokenizer(input_text, return_tensors='pt')# Apply dynamic quantization to the modelquantized_model = torch.quantization.quantize_dynamic( model, {torch.nn.Linear}, # Specify the layers to quantize dtype=torch.qint8 # Specify the target data type)# Verify that the model is quantizedprint(quantized_model)# Move the model to the appropriate device (CPU)device = torch.device('cpu')quantized_model.to(device)# Perform inferencewith torch.no_grad(): outputs = quantized_model(**inputs)# Print the output tensorprint(outputs.last_hidden_state) In this example, we applied dynamic quantization to the linear layers of the DistilBERT model with the method torch.quantization.quantize_dynamic(), converting them to 8-bit integers (qint8): this reduces the model size and improves inference speed on supported hardware.

Strategy 4 to Keep Self-hosted LLM Costs Down: Effective Data ManagementEfficient data management can minimize storage and processing costs. This is particularly useful when using LLMs, because, since these models require a huge amount of data, effectively managing them can significantly decrease associated costs.

Here are some best practices to do so:

• Data preprocessing. To reduce the amount of data an LLM needs to process – thus, saving on hardware resources – a best practice is to preprocess the data before feeding the model with it. In the case of LLMs, you can generally follow these steps for preprocessing data:
  1. Data cleaning. Remove noise like special characters, HTML tags, and irrelevant content.
  2. Handle Missing Data. Address gaps in the dataset through imputation or removal.
  3. Normalization. Standardize text by converting to lowercase, handling contractions, and standardizing punctuation.
  4. Tokenization. Split text into meaningful tokens, like words or subwords.
  5. Stop words removal. Eliminating the common words that add little value to the text (e.g., “the,” “and”).
  6. Data balancing. Ensure diverse and representative data to prevent model bias.
  7. Encoding. Convert text into numerical format using techniques like word embeddings or token IDs.
• Caching. Caching temporarily stores copies of data to improve access speed and reduce latency. By keeping frequently accessed data in a high-speed storage layer (usually RAM), caching minimizes the need to repeatedly fetch data from slower storage or remote servers. This boosts performance and efficiency, particularly for repeated read operations. So, in the context of LLMs, to reduce the need to repeatedly process or access the same data, you can implement a caching system for frequently accessed or processed data; this will save on computational resources.
• Data lakes and warehouses. As LLMs need to ingest and process large quantities of data, you should retrieve this data somewhere. Today, data lakes, data warehouses, and data lakehouses (like Snowflake or Databricks, just to name a few) allow you to store and manage large volumes of data efficiently, at low fees. Well, indeed it’s more cost-effective to use them than to store data on your hardware. To get an idea, you can see Snowflake and Databricks pricing models.

ConclusionsIn this article, we’ve discussed four methodologies to keep self-hosted LLM costs down.

The important concept to bear in mind is that these methodologies can be combined together to provide even better results in saving costs.

So, for example, given your needs and infrastructure, you can decide to containerize a pruned LLM, while implementing batching techniques and a caching system.

The post Keeping Self-Hosted LLM Costs Down: Best Practices and Tips appeared first on Semaphore.

View Details

During the development phase of a project, there are some common measures you’d typically take to ensure your code is at industrial standard. These steps include checking if it’s secure or contains not-so-obvious vulnerabilities and ensuring it’s easily compatible with other code.

You can take these steps manually through peer review and other processes. However, this method could lead to more problems or unresolved issues. Automating this procedure will be the best approach for your team, and you can achieve this using static code analysis tools.

Static code analysis tools allow you to analyze your source code without running it. They examine the code for potential issues, vulnerabilities, or violations of coding standards. Some examples of these tools include Coverity, CodeScene, veracode and the focus of this article, SonarQube.

This article lets you understand SonarQube, its features, and why you need it for your project. You’ll get to install SonarQube and have a quick overview of its console. By the end of this article, you’ll gain more confidence in the quality of your code.

Understanding SonarQubeSonarQube is an open-source static code analysis tool. It allows you to continuously monitor, analyze, and improve your code. After your code analysis, SonarQube provides a report with the results.

This report provides detailed metrics to help you understand and improve the quality of your codebase over time. Additionally, it offers recommendations to improve your codebase.

SonarQube is an overall quality management tool. This means it not only does code analysis but also code coverage and a report-generating system for any test you carry out, like unit tests.

Although it’s a Java-based tool, it’s not limited to Java. It can analyze 30+ programming languages and infrastructure as code (IaC) platforms.

SonarQube offers you two key features. The first is the Quality Gate, and the second is the Customizable Ruleset. Quality Gates in SonarQube act as checkpoints in your CI/CD pipeline, ensuring that your code meets predefined quality criteria before it can be merged into the main branch or deployed.

Customizable Rulesets in SonarQube allow you to define specific coding rules and standards tailored to your project’s requirements and guidelines. SonarQube uses these rules during static code analysis to identify issues such as bugs, code smells, and security vulnerabilities.

Why You Need SonarQubeIf you understood the section above, you’ll notice that SonarQube offers you so much. Well, that’s just a bit of what you can do with this tool. The following are some reasons why you should use SonarQube:

– Cost savings: No organization loves to waste money. By identifying and fixing issues early in the development cycle, SonarQube can help reduce the cost of software maintenance and support over time.

– Easy CI/CD integration: SonarQube integrates seamlessly with CI/CD pipelines, allowing you to automate code analysis as part of your development workflow. This helps catch issues early and ensures that only quality code is merged and deployed.

– Confidence in your code quality: If your project needs to adhere to specific coding standards, industry regulations, or security requirements, SonarQube can help enforce these standards, making you more confident about the quality of your codebase.

– Security Assurance: With its built-in security hotspots feature, SonarQube helps identify and mitigate security risks in your codebase, protecting your software from potential threats and attacks.

Setting Up SonarQubeThere are several ways to set up a SonarQube server. One is downloading the zip file from the SonarQube website. The other uses the community edition Docker image. In this tutorial, you’ll be using the latter.

PrerequisitesTo install the SonarQube server, ensure you have Docker installed on your machine. Aside from that, you need the following hardware and software requirements:

• At least 4GB of RAM for small-scale installations and 16GB of RAM for large ones.
• You’ll need a 64-bit operating system with at least two cores for small-scale installations and eight for large ones.
• At least 30GB of disk space for small-scale installations
• You’ll need a web browser to access the SonarQube web interface.

For a complete list of hardware and software requirements, please refer to the SonarQube installation requirements page.

The following steps will guide you in installing the SonarQube server using Docker. This method is more straightforward to set up, and you won’t need to install any additional software like Java.

Step 1: Run the Docker imageOnce you have installed Docker on your machine, you can run the Docker image on your command line.

docker run -d --name sonarqube -e SONAR_ES_BOOTSTRAP_CHECKS_DISABLE=true -p 9000:9000 sonarqube:latest This command pulls and runs the latest version of SonarQube on your machine. You can confirm that it’s running by checking the Docker status.

Figure 1: Docker StatusStep 2: Access the SonarQube web interfaceAfter the installation, it takes a few minutes for the SonarQube server to start. Once it starts, you can access the SonarQube web interface at http://localhost:9000.

Figure 2: SonarQube startup pageStep 3: Configure SonarQubeYou’ll be prompted to set a login and password for the SonarQube server. The default login and password is admin for both.

You’ll be prompted to change the default password once you’ve logged in using the default credentials.

Figure 3: Change SonarQube default settingsInput the new password and confirm it. Once you’re done, you can reaccess the SonarQube web interface.

Figure 4: SonarQube ConsoleQuick Overview of the SonarQube ConsoleTo have a good overview of the console, we’ll set up project and analyze it using SonarQube.

Step 1: Select the project sourceYou can create a new project using the Projects section of the SonarQube console. This section determines where the project source is located. This can be on your local machine or a remote repository like GitHub or GitLab.

In this guide, you’ll create a local project. To do that, go to the Projects section of the SonarQube console and click Create a local project.

Figure 5: Create a new projectStep 2: Add project descriptionAfter selecting the local project, you will be prompted to provide a Project display name, Project key and Main branch name. The project display name and project key** are used to identify the project in the SonarQube console.

The Main branch name is the name of the main branch in the project. The default value is main.

Figure 6: Add project descriptionStep 3: Setup Clean as You CodeClean as You Code (CaYC) is a SonarQube methodology for software development that allows you to take responsibility for the quality of the code you produce, mainly focusing on the new code that you’re adding or modifying.

In this approach, you’ll aim to write clean, readable, and maintainable code from the beginning rather than postponing cleanup or refactoring to a later time.

You’ll be prompted to select your Clean as You Code option. You have the following options:

• Previous Version: Considers any code that has changed since the project’s most recent version.
• Number of Days: Specifies a floating new code period based on a specified number of days.
• Reference Branch: Considers changes made between your branch and a specific reference branch as new code.

For this example, we’ll choose Previous Version.

Figure 7: Configure Clean as You CodeAfter this, click on Create project**, and SonarQube will create your project in the console.

Figure 8: Project createdStep 4: Select Analysis MethodAfter the project is created, you’ll be prompted to select the Analysis method**. The Analysis method lets you specify how you want to analyze your code.

The options are CI/CD tools such as GitHub Actions, Jenkins, SemaphoreCI, etc. For this example, we’ll choose Locally.

Figure 9: Select Analysis MethodStep 5: Set up a project tokenYou’ll be prompted to enter a Project token**. This token is used to authenticate with SonarQube. This token will notify you when you perform an analysis.

Add the project name and click the Generate button to generate the token.

Figure 10: Project tokenAfter which, you can copy this token and click the Continue** button.

Figure 11: Copy tokenStep 6: Run analysis on the projectAfter creating the token, you can now analyze the project. You’ll need to select the format that best fits your build. This can be either Maven, Gradle or for any programming language like JS, Python, Go**, etc.

After which, you’ll need to download the SonarQube Scanner to your local machine – it, in turn, will be running against the SonarQube server we’re running via docker at http://localhost:9000. The SonarQube Scanner is a tool that automatically runs analysis on your code. You can download the SonarQube Scanner from the SonarQube Scanner download page.

You’ll need to configure your project once you’ve downloaded the SonarQube Scanner that fits your OS and architecture.

NOTE: There is no support for the ARM64 architecture. To bypass this, install the JRE and choose the Any option on the installation page.

In the root directory of your project, create a sonar-project.properties file and add the following properties:

```

must be unique in a given SonarQube instancesonar.projectKey=Test-project# --- optional properties ---# defaults to project key#sonar.projectName=My project# defaults to 'not provided'#sonar.projectVersion=1.0 # Path is relative to the sonar-project.properties file. Defaults to.#sonar.sources=. # Encoding of the source code. Default is default system encodingsonar.sourceEncoding=UTF-8

`` Replace thesonar.projectKey` with your project key.

After this, on your SonarQube console, you’ll see a prefilled sonar-scanner command containing your project details. This command should look like this:

sonar-scanner \ -Dsonar.projectKey=Test-project \ -Dsonar.sources=. \ -Dsonar.host.url=http://localhost:9000 \ -Dsonar.token=sqp_448617bdbf2dd550b03fbfffcae92b9011e6f35b Copy the command, navigate to your project directory via the command line and run it.

Figure 12: SonarQube ScannerOnce your execution is successful, as in the image above, switch back to the console. You should now see the analysis results in the SonarQube console.

Figure 13*: SuccessNOTE*: The project used for this tutorial is a local JavaScript project. You can use any project of your choice, but if you prefer, you can clone this one on GitHub. Since it’s a JavaScript project, you’ll need Node Js installed on your machine for the Sonar scanner to work.

Step 7: Analyze the ReportOnce you get back to the console, you’ll see your analysis report. This report is quite detailed and can get confusing, so let’s go through each section.

Figure 14: SonarQube Console Projects SectionThis page contains overall information about the project. On this page, you can see whether your scan succeeded or failed. You can also find the scores of each metric.

These metrics are security, reliability, maintainability, hotspots, coverage and duplications. Scrolling down the page will give you more details, like the size and languages used in the project.

Figure 15: SonarQube Console IssuesIn the issues section, you can see the issues that were found in every file of your project. Under each file, you can find the year the file was created and assign an open issue to a team member.

Figure 16: SonarQube Console RulesThe rules section, as the name implies, contains rules that are used to identify security issues. They are categorized into four types: code smell, bug, vulnerability, and security hotspot.

Each type has specific expectations, such as zero false positives for bugs and code smells and more than 80% true positives for vulnerabilities. Rules can be filtered by language, type, tag, repository, severity, status, etc.

Figure 17: SonarQube Quality ProfilesA quality profile defines the rules and configurations you can apply during code analysis for a specific programming language. It determines the coding standards, best practices, potential bugs, security vulnerabilities, and code smells checked for in the codebase.

You can customize or extend a quality profile to meet your project’s requirements so the quality of your code is consistent across projects.

Figure 18: SonarQube Quality GateA quality gate allows you to define a set of conditions that must be met before a project can be released. They define criteria like bug counts or code coverage.

The default quality gate is “Sonar way”; however, you can create your own. Permissions ensure that only authorized users can edit or manage quality gates. Only users with global administration permissions can modify quality gates by default, but this permission can be delegated to specific experts or groups for individual gates.

Figure 19: SonarQube AdministrationThe administration section allows you to configure your console. Here, you can authenticate with DevOps platforms like GitLab, GitHub, etc.

You can also set general rules concerning your project creation, quality profiles, and the Look & feel of your console.

ConclusionIn this article, you learnt how to analyze a project using SonarQube. In the process, you understood how to use this tool, installed the SonarQube server, and had a quick overview of the console.

All this information might be confusing initially; however, once you master the console, maintaining clean, secure and reusable code will be easy.

Once you’ve gotten the hang of this, you should look out for the next article, in which we’ll go through how to integrate SonarQube into your Semaphore CI pipeline.

The post SonarQube: Open Source Security Automation appeared first on Semaphore.

View Details

In today’s time, many organizations are using cloud technologies. Businesses rely heavily on cloud services for their day-to-day operations. And managing cloud expenses has become more critical than ever. But, with the growing adoption of cloud services, optimizing their costs has become a critical challenge.

According to Gartner’s report of November 2023, global spending on public cloud services is forecast to grow 20.4% to a total of $678.8 billion in 2024, up from $563.6 billion in 2023. As organizations scale up their operations, the cloud costs can go out of control if not managed effectively. Fortunately, many open-source tools are available in today’s time. These tools can help mitigate these costs while ensuring optimal performance and resource use.

Importance of reducing cloud expensesWhen organizations adopt cloud computing solutions they often encounter the task of overseeing and managing cloud expenses. Failure to optimize the use of cloud resources can result in overspending which impacts an organization’s performance significantly. The pay-as-you-go approach in cloud services requires monitoring and optimization to ensure proper resource usage and cost control. Neglecting this model could potentially result in setbacks, for the organization. Let’s explore the benefits that businesses can derive from optimizing their cloud spending:

• Cost Efficiency: By identifying unnecessary spending, organizations can divide their resources more efficiently.
• Great Budget Management: Predictable cloud expenses can lead to great budget management.
• Increased Return-on-investment (ROI): Having good control over cloud resources can lead to an increased ROI.

Now that we understand the benefits & importance, let’s see the 10 open-source tools that will help an organization achieve these cloud cost optimizations.

Overview of open-source tools as cost-saving solutionsThe open-source community has stepped up to offer a mass of powerful and cost-effective solutions. These tools can reveal the path to optimized cloud expenses, as they provide organizations with the necessary insights, automation, and control to navigate the cloud landscape.

Here are the top open-source cloud cost optimization tools, that will help you achieve your goals.

Tool 1: KubernetesKubernetes also known as K8s is a dominant open-source container orchestration platform. It is renowned for its ability to automate the deployment, scaling, and management of containerized applications. It works by simplifying the containerized workloads based on demand, across a cluster of machines to ensure increased availability, scalability, and efficient resource use. We may also call this a container orchestration powerhouse tool.

Cost-saving features of KubernetesThere are numerous features of Kubernetes for cost saving in the cloud:

1. Autoscaling: Kubernetes consists of some open-source tools (like Horizontal Pod Autoscaler and Cluster Autoscale) that allow the users to dynamically adjust the number of containers to use which prevents the companies from overprovisioning, where they pay for resources that are not being used.
2. Improved developer efficiency: This also improves the developer efficiency by streamlining deployments, rollbacks, and scaling allowing them to focus on building apps rather than managing cloud infrastructure.
3. Horizontal scaling: It also gives the users leverage the horizontal scaling, allowing them to distribute workloads across multiple nodes, optimizing resource usage and reducing costs.

Real-World examplesSome companies such as Netflix, Spotify, Pearson, etc. use Kubernetes. These companies used Kubernetes efficiently for their containerized microservices, autoscaling, and other services, and also reduced their cloud infrastructure cost by a large percentage.

Kubernetes GitHubTool 2: TerraformTerraform is better called an infrastructure-as-code tool that helps define your infrastructure resources (like servers, storage, and networks) in declarative configuration files, enabling automation, repeatability, and cost optimization. Terraform has multiple use cases including infrastructure-as-code, managing Kubernetes, managing virtual images, managing network infrastructure, etc.

Note: As of November 2023, Terraform is no longer an open-source project due to the acquisition of HashiCorp by IBM for $6.4 billion, as reported by ycombinator news. If you’re seeking a fully open-source option for the cloud cost optimization, OpenTofu is a promising alternative.

Terraform as a cost-effective infrastructure tool1. Infrastructure Optimization: * Resource Visibility: Terraform provides a clear overview of your entire infrastructure, making it easier to identify and eliminate unused or underutilized resources. * Right-Sizing: You can define resource configurations with the exact specifications your applications require, preventing overprovisioning. 2. Reduced Errors & Faster Configuration: With Terraform, you can automate the whole process, minimizing costly mistakes. Also, the infrastructure changes are applied quickly and consistently through code which saves time and reduces errors. 3. Cloud Cost Management: This tool integration in your application can also provide the early cost estimation of infrastructure based on your configurations. This allows you to make informed decisions about resource types and sizes before provisioning. Not only this, automated checks can also be applied to set it within the budget limits.

Real-World examplesSome companies such as Deutsche Bank and GitHub use Terraform to simplify infrastructure provisioning across multiple cloud providers, leading to greater efficiency and cost savings.

Terraform GitHubTool 3: GrafanaGrafana is an open-source monitoring and observability system that provides powerful visualization capabilities for analyzing and understanding data from various sources, including cloud services

Grafana as Cost Efficiency ToolNow let’s see, how grafana helps in identifying cost inefficiencies and optimizing resource usage.

1. Visualize Cloud Costs: Integrate Grafana with cloud providers (AWS, Azure, GCP) to ingest cost data. Create custom dashboards that display cost metrics like hourly, daily, or monthly spending by service, project, or region. Utilize heatmaps and other visualizations to identify cost spikes, trends, and anomalies.
2. Drill Down for Insights: Grafana allows you to drill down into specific cost components, such as CPU, storage, network usage, or database instances. Correlate cost data with other infrastructure metrics (e.g., CPU utilization) to understand the root cause of cost variations. This helps you identify idle resources or inefficient configurations that are driving up costs.
3. Set Cost Alerts: Configure alerts that trigger when spending exceeds predefined thresholds or when anomalous cost patterns arise. Early detection of potential overspending allows you to take corrective actions, such as scaling down resources or optimizing configurations.

Real-World examplesThe companies that use AWS can actually use the grafana for getting the visualizations by creating their personalized dashboards that will display:

• Total daily spending on AWS services.
• Cost breakdown for EC2 instances, S3 storage, and Lambda functions.

While Grafana is a powerful open-source tool, it requires setting up data sources and configuring dashboards to get a detailed cost analysis, effective cost monitoring, and optimization. Here are some of the dashboards available online, to get you started quickly, that specifically focus on cost monitoring:

1. Amazon Managed Grafana for Cost Anomaly Detection
2. Kubernetes Dashboard for cost management

Grafana GitHubTool 4: PrometheusPrometheus is another open-source tool that is designed to work as a monitoring and alerting toolkit, especially for cloud-native platforms. It collects and analyzes the metrics from your cloud infrastructure, enabling you to monitor resource utilization and reduce costs.

How Prometheus assists in monitoring cloud resource utilization and identifying cost-saving opportunitiesPrometheus is a powerful real-time monitoring tool that enables alerting capabilities making the one of the reasons to effectively utilize it for cost optimization. There are several features that Prometheus provides including:

1. Resource Consumption Insights: With the help of this tool, the user can understand how efficiently our resources are being utilized by taking into consideration different metrics like CPU, memory, and storage usage for cloud resources.
2. Identifying Idle Resources: Another feature of using Prometheus is that we can easily identify resources with consistently low utilization rates, indicating potential overprovisioning.
3. Cost Correlation: Combine Prometheus metrics with cost data from your cloud provider. This correlation can help you understand which resources are driving up costs and how changes in resource usage affect your cloud bill.

Examples of Prometheus in reducing cloud expensesThere are various examples of using Prometheus to monitor the EC2 instances. With this tool, companies discovered that 20% of instances have consistently low CPU utilization. They right-size these instances to a smaller size, saving on compute costs. SoundCloud (the popular audio streaming platform) is one of the good examples that uses Prometheus for monitoring and cost optimization.

Prometheus GitHubTool 5: Apache KafkaApache Kafka is another open-source tool developed by LinkedIn mainly for distributed data streaming. Apache might not be the most be best fit for cost-saving tools as compared to those discussed above, but it offers several features that can significantly reduce cloud infrastructure expenses specifically when handling large volumes of online data streams. It is being used by more than 80% of all Fortune 100 companies.

Cost-saving features of Kafka, such as efficient message processing and storageApache Kafka has some of its top core capabilities like high throughput, scalable, high availability, etc. But Apache Kafka also offers several features that can actually reduce the cloud infrastructure cost, when handling large amounts of data streams.

There are several features that Apache Kafka provides like:

1. Efficient message processing: Kafka acts as a buffer between data producers and consumers which is known as decoupling which allows producers to send data at their own pace without affecting consumers and vice versa and finally, reduces the resource consumption on both sides, leading to potential cost savings.
2. Flexible storage options: Log compaction in Kafka reduces storage costs by removing duplicates and retaining only the latest version, optimizing long-term data retention. Additionally, tiered storage options like Confluent Cloud minimize expenses by offloading older data segments to cheaper cloud object storage.
3. Resource optimization: Apache Kafka allows the dynamic scale of the clusters in the cloud based on your data volume and processing needs allowing you to prevent overprovisioning and reducing the idle resources cost.

Real-World examplesLinkedIn being the innovator of this tool is the best example of using Apache Kafka to optimize its data processing workflows and reduce cloud expenses.

Apache Kafka GitHubTool 6: ElasticsearchElasticsearch is a distributed search and analytics tool designed to handle large volumes of data efficiently. It is an open-source tool that organizations use to manage and analyze datasets cost-effectively.

How Elasticsearch helps in managing and analyzing large volumes of data cost-effectivelyElastisearch is a free and open-source tool that can help the organization manage and analyze large volumes of data but not all services are free, some need to be paid to use. Reducing the cost with elastisearch can be done by:

1. Efficient Data Storage: Using the elastisearch’s inverted indexes for optimizing the searches for specific terms reduces the data scanned along with the lower resource consumption. Another tool Elasticsearch ILM (Index Lifecycle Management) automates the process of moving older data to cheaper storage tiers which helps you optimize storage costs for data with varying access needs.
2. Horizontal Scaling: Elasticsearch clusters can be scaled horizontally by adding more nodes, allowing you to handle growing data volumes without sacrificing performance. This eliminates the need for expensive hardware upgrades.
3. Search Appliance Alternative: Elasticsearch can replace traditional search appliances, which can be expensive and have limited scalability.
4. Use data rollups: Summarize data into a single document, then archive or delete the original data.

Examples of Elasticsearch use cases for reducing cloud costsElastisearch is a widely popular and adopted open-source tool that helps many organizations leverage its cost-saving features like data storage, horizontal scaling, etc.

For example, a company that uses Elasticsearch to store and analyze website logs, and if they are implementing ILM, they can easily move older logs to cheaper cold storage, significantly reducing storage costs. Similarly, if a company leverages Elasticsearch to power its e-commerce search engine. The efficient storage and indexing capabilities of Elasticsearch, enable them to handle large product catalogs and high search volumes without incurring high infrastructure expenses.

For those on the AWS cloud platform, AWS OpenSearch Service provides a managed offering with similar functionalities and potential cost-saving benefits.

Elastic GitHubTool 7: HadoopHadoop is a Java-based distributed processing framework that is used for big data analytics and processing, not only this, but its scalable and fault-tolerant architecture enables the consumer to reduce the cost of resources running in the cloud.

Cost-saving benefits of Hadoop for data processing and storage in the cloudOrganizations can benefit from using Hadoop to optimize data workflows, minimize data movement costs, and efficiently process large datasets. As it is open-source, it’s free but that doesn’t mean, all of the services are free. The cost will depend on how much the software is being used and at which scale.

Reducing the cost with Hadoop can be done like:

1. Scalability: The clusters in Hadoop are easily scalable up or down both ways and its tools allow you to pay only for the resources you use, especially in cloud environments with on-demand pricing models like YARN (Yet Another Resource Negotiator).
2. Open-Source: Like others, Hadoop also discards the licensing costs associated with proprietary data processing solutions.
3. Data Lake Architecture: The data lake architecture in Hadoop has the ability to store large volumes of structured, semi-structured, and unstructured data in a centralized data lake, and this can separate data silos and simplifies data management, potentially reducing storage costs.

Examples of Hadoop implementations for cloud cost reductionSeveral companies use Hadoop services like, Adobe, LinkedIn, Facebook, etc. To have a real-life implementation for cloud cost reduction, an organization can use a cloud-based Hadoop cluster to process vast amounts of user activity data and social media sentiment. With this technique, the organization can reduce the on-demand pricing and scale the cluster based on daily/weekly usage patterns, they optimize their cloud spending.

Hadoop GitHubTool 8: OpenStackOpenStack is an open-source cloud computing platform that allows organizations to create and manage their own private clouds. This tool helps organizations optimize cloud expenses through efficient resource utilization, OpenStack becomes a flexible and cost-optimization alternative tool to other paid cloud tools.

How OpenStack enables cost-effective private cloud deployments and managementOpenStack can enable the organization for cost-effective private cloud deployments, the user can optimize some of the below features:

1. Compute (Nova): OpenStack’s Compute service allows users to create and manage virtual machines, providing an alternative to proprietary compute services like Amazon EC2 or Google Compute Engine. By utilizing Nova, organizations can potentially reduce costs associated with virtual machine provisioning and management.
2. OpenStack’s Swift: Swift for object storage and Cinder for block storage which offer alternatives to proprietary storage solutions such as Amazon S3 or GCP storage. By using Swift and Cinder, organizations can avoid vendor lock-in and potentially reduce storage costs.
3. The Distributed Resource Scheduler (DRS): DRS is another feature that distributes workloads across multiple nodes based on resource usage. Balancing workloads optimizes resource usage, reducing costs.
4. Use Open-Source Tools: OpenStack is an open-source platform. Consequently, you can use open-source tools to monitor and manage your infrastructure. The open-source tools in OpenStack give organizations detailed control over large volumes of resources including computing, storage, etc.

Examples leveraging OpenStack for cloud expense optimizationOpenStack is being used by various big organizations such as IBM, Walmart, VMware, NASA, etc. However, it’s important to acknowledge the trend of organizations adopting a hybrid approach, integrating OpenStack with Kubernetes (k8s). This combination leverages the strengths of both platforms: OpenStack’s infrastructure management capabilities and k8s’ container orchestration expertise. The users can integrate the openstack-integrator to be able to use the OpenStack native features.

Organizations can benefit the cost reduction with this tool with various features like one, by dynamically scaling the OpenStack cluster based on project requirements, optimizing resource utilization, and avoiding paying for idle resources in a public cloud environment. Another best cloud cost optimization can be done by using a distributed resource scheduler (DRS), reserve tools, and more.

OpenStack is very useful for specific use cases where there are predictable workloads, strict security requirements, or fluctuating computational needs, and with this, it provides flexibility, neutrality, and the potential for significant cost savings in the long term for an organization.

OpenStack GitHubTool 9: DockerDocker is an open-source containerization platform consisting of a variety of components that help in container management. Docker can also become a great tool to reduce and optimize cloud expenses, by optimizing resource utilization and streamlining deployments.

But why use Docker to optimize the cloud expenses in today’s time? The answer is simple, docker allows you to package your applications and their dependencies into lightweight, portable containers. These containers share the underlying operating system of the host machine, eliminating the need for full virtual machines (VMs) and thereby monitoring the control of extra pay for services.

How Docker facilitates efficient resource utilization and deployment in the cloudThere are several reasons why docker can facilitate efficient resource utilization and deployment in the cloud and can lead to cost-saving advantages:

1. Reduced Resource Consumption: Containers require fewer resources compared to VMs. This translates to lower compute, memory, and storage costs in the cloud. By packing more containers onto fewer servers, you can optimize resource utilization and potentially reduce the number of instances needed.
2. Faster Deployments: Docker containers are self-contained and boot up quickly, accelerating application deployment and scaling processes. This minimizes the time applications spend in a non-productive state, reducing associated cloud costs.
3. Simplified Management: Docker facilitates consistent deployments across development, testing, and production environments. This reduces the need for separate infrastructure for each stage, potentially saving on cloud resources.
4. Microservices Architecture: Docker’s containerization model promotes microservices architecture, where applications are broken down into smaller, independent services. This allows for easier scaling and resource allocation based on individual service needs, further optimizing cloud costs.

Examples of Docker for lowering cloud expensesDocker nowadays has become a very popular tool for containerization for many organizations and for some to lower the cloud cost too. Big organizations like Google, AWS, Thoughtworks, etc. are using Docker in their tech stack.

Organizations can leverage Docker to optimize cloud expenses by reducing the overhead associated with VMs and improving deployment efficiency. For example, a company that uses docker containers for its microservices architecture, by scaling individual services based on traffic patterns, can optimize resource utilization and reduce cloud costs compared to running monolithic applications on VMs.

Another alternate tool that organizations can leverage for cost saving is Docker Swarm which is a native container orchestration platform from Docker. It allows for efficient management of containerized applications at scale, load balancing, etc.

Docker GitHubTool 10: Apache SparkApache Spark is an open-source distributed data processing multi-engine that can execute data engineering, data science, and machine learning on single-node machines or clusters. Spark is the best tool for handling the large amount of datasets efficiently. Not only this, it can actually reduce the cloud costs for big data analytics and processing.

Cost-saving features of Spark for big data analytics and processing in the cloudSpark is a powerhouse for distributed data processing where data is processed in parallel across clusters of machines, handling complex data tasks faster as compared to other approaches. But this isn’t its cost-saving feature, it also includes:

1. Resource optimization: To ensure efficient resource utilization and avoid overprovisioning of compute resources, we can configure resource allocation for different stages of your data processing jobs with the usage of tools like Kubernetes.
2. Unified Platform: As Spark contains various big data tasks, including batch processing, real-time streaming, and machine learning, we can hide the need for separate tools and associated infrastructure costs.
3. In-memory processing: Cost can be reduced like storage costs and faster processing times, allowing you to complete tasks quicker and potentially reduce the number of compute instances needed. Spark can leverage in-memory computing for frequently accessed data, significantly reducing disk I/O operations.

Examples of cost reduction using Apache SparkApache Spark is the most widely used engine for scalable computing. It is being used by thousands of organizations, including 80% of the Fortune 500 like Databricks, Yahoo, Netflix, and more.

For example, if an organization uses Spark’s in-memory processing for frequently accessed data, it can achieve faster turnaround times and reduce storage costs compared to traditional disk-based analytics to analyze large datasets of customer transactions for a finance organization. Similarly, if an organization can handle complex computations efficiently and faster, it can lead to cost savings.

Apache Spark GitHubConclusionOptimizing cloud expenses is crucial for businesses striving for long-term financial sustainability. However, it’s a multifaceted challenge. In this article, we explored 10 open-source tools – Kubernetes, Terraform, Grafana, Prometheus, Apache Kafka, Elasticsearch, Hadoop, OpenStack, Docker, and Apache Spark that can reduce cloud costs and may be chosen as cost optimization tools. Additionally, tools like Semaphore CI can be leveraged to streamline and automate cloud deployments, further contributing to cost optimization. For a comprehensive CI/CD solution that streamlines deployments, explore Semaphore CI Cloud. These tools smoothen the operations, minimize wastage, and ensure significant savings in cloud deployments.

The post 10 Open-Source Tools for Optimizing Cloud Expenses appeared first on Semaphore.

View Details

Cloud storage has become the most popular way for businesses and developers to store data for online use. S3 is the de facto standard for cloud storage, with Amazon as the leading provider. However, the monthly bill for Amazon S3 can be steep, often amounting to thousands of dollars per month, mainly due to data transfer (egress) fees. Enter Cloudflare R2, a cost-effective cloud storage alternative that eliminates these greedy egress fees!

In this article, you will understand what Cloudflare R2 is, how it compares to Amazon S3, how its pricing model works, and whether you can migrate your data from S3 to R2.

Let’s dive in!

What Is Cloudflare R2?R2 is a modern, efficient, S3-compatible cloud object service offered by Cloudflare. Compared to typical cloud storage services, such as Amazon S3, it is a more cost-effective solution. That is because Cloudflare R2 allows storing large amounts of unstructured data without costly egress bandwidth fees.

Data egress fees are network fees that cloud providers typically charge for moving data out of the cloud. These, also known as data transfer fees, are usually hard to predict and add up to the fees you already need to pay for storage and cloud computing. Egress fees apply when retrieving files from cloud storage (e.g., to show images on a web page), accessing data from external analytics applications, or transferring data between different regions.

Other scenarios where companies need to move data out of their cloud storage is when implementing hybrid cloud (a mix of two or more cloud environments) or multi-cloud (multiple clouds from more than one cloud provider) architectures. These enable businesses to use the most appropriate cloud environment for each task, improving the efficiency and flexibility of the overall IT infrastructure. In addition, having more than one cloud storage is essential for robust disaster recovery.

By eliminating egress fees, R2 makes it more affordable for businesses and developers to adopt such robust, reliable, and efficient IT architectures. In particular, Cloudflare R2 covers several scenarios and use cases, including:

• Cloud storage for web applications
• Storage for multimedia files (e.g., images, podcast episodes, recordings, etc.)
• Data lakes (e.g., for analytics and Big Data applications)
• Storage for cloud-native applications
• Cloud storage for large batch processes (e.g., machine learning model artifacts or datasets)

AWS S3 vs Cloudflare R2Amazon S3, short for Amazon Simple Storage Service, is a scalable object storage service launched by AWS in 2006. Over the years, the S3 standard has become the go-to API for cloud storage. Its popularity has led to the emergence of several S3-compatible providers, but Amazon remains the S3 cloud storage leader.

Cloudflare R2 is a much newer object storage solution, with its first release only in 2021. It offers similar features to Amazon S3, relying on an S3-compatible API. This makes it easy for users to migrate and integrate their existing tools and applications with this cloud system.

The main differences between Amazon S3 and Cloudflare R2 are:

• Cost-effectiveness: Amazon S3 charges high egress fees for moving data off the AWS network, which can significantly increase overall cloud-associated costs. In contrast, Cloudflare R2 has no egress fees. Users can move data out of storage at no additional cost, resulting in significant savings in data-intensive applications.
• Free tier: Amazon S3 offers a limited free tier, while Cloudflare R2 comes with a generous free tier that includes millions of requests per month.
• Global accessibility and performance: Amazon S3 provides solid performance, but only when configured correctly to optimize data transfer between regions. On the other hand, Cloudflare R2 automatically selects the optimal storage region based on the data upload location. This ensures high performance and availability without the need for complex configurations.
• Cloud storage API: Amazon S3 is based on the original S3 standard, which is widely supported by a wide range of tools and applications. Instead, Cloudflare R2 uses an S3-compatible API, which lacks some S3 features but has others. Full compatibility with the S3 API for all features is currently underway. Monitor the status on the official docs.
• Data security: The two solutions offer strong security features, including encryption and in-depth access control policies. Cloudflare R2 enhances security even more with undiscoverable bucket names, reducing the risk of unauthorized access.

Cloudflare R2 Pricing BreakdownThe Cloudflare R2 pricing model consists of two following dimensions:

• Storage: Total volume of data stored in the cloud, per GB.
• Operations: Operations performed on the data in the cloud. These are divided into Class A and Class B. Class A operations are more expensive and tend to change the state, while Class B operations are cheaper and tend to read the existing state. Some operations, such as deleting an object or a bucket, do not fall into these two categories and are free.

The R2 pricing can be summarized in the table below:

| Rates | | --- | | Storage | $0.015/GB | | Class A operations | $4.50/million requests | | Class B operations | $0.36/million requests | | Data egress | Free |

All included usage is charged on a monthly basis.

Is this actually more cost-effective than Amazon S3 storage? Let’s find it out!

Cloudflare R2 vs Amazon S3: Price ComparisonTake a look at the summary tables below to compare the free and paid plans offered by Cloudflare R2 and Amazon S3:

| Cloudflare R2 | Amazon S3 | | --- | --- | | Data storage | $0.015 per GB | Starts at $0.023 per GB | | Class A operations (POST) | $0.0045 per 1,000 requests | $0.005 per 1,000 requests | | Class B operations (GET) | $0.00036 per 1,000 requests | $0.0004 per 1,000 requests | | Data egress | Free | Starts at $0.09 per GB |

Note that prices are up-to-date at the time of writing but may change over time. Amazon S3 prices and egress fees are based on tiered rates for S3 Standard in the US East region. For more information, visit the Amazon S3 pricing page.

As you can see, Cloudflare R2 is cheaper in all the considered aspects when compared to Amazon S3. Time to see how these price variations translate to a couple of real-world scenarios!

Free Tier LimitsBoth Cloudflare R2 and Amazon S3 come with free tiers, which compare as below:

| Cloudflare R2 | Amazon S3 | | --- | --- | | Data storage | 10 GB per month | 5 GB per month | | Class A operations (POST) | 1M per month | 2k per month | | Class B operations (GET) | 10M per month | 20k per month | | Data transfer | Unlimited | 100 GB | | Expiration | Unlimited | 12 months |

Notice how the biggest difference is the huge amount of free requests that R2 offers. Millions of Class A and Class B operations can meet the cloud needs of a medium-sized web application. Data storage limits are quite restrictive in both scenarios, but R2 offers twice as much monthly storage space as Amazon S3. Plus, Amazon S3’s free tier expires after 12 months, while R2’s has no time limit.

To better understand what you can achieve with a free tier, let’s consider an example. With Cloudflare R2’s free tier, you can develop a site where users upload up to 10,000 1 MB web-optimized images per month. Suppose each web page on the site contains 5 images, this means you could have 2,000,000 monthly visits without incurring any costs. With Amazon S3, you could only have 5,000 uploaded images and 4,000 views a month.

Usage for Video StorageLet’s assume that you are building a video-oriented application. Users will upload 1 million videos per month and view 100 million of these videos in this application. The average size of the video files is 5 MB, which means that users will upload 5 TB of storage per month and download 500 TB of data a month.

To compare the cost of this application on R2 and S3, it is necessary to dig into the Amazon S3 prices. In detail, the S3 pricing model for data storage follows this table:

| S3 Data Storage | Cost | | --- | --- | | First 50 TB/month | $0.023 per GB | | Next 450 TB/month | $0.022 per GB | | Over 500 TB/month | $0.021 per GB |

Similarly, S3 data transfer fees are calculated using the following table:

| S3 Data Transfer Tier | Cost | | --- | --- | | First 10 TB/month | $0.09 per GB | | Next 40 TB/month | $0.085 per GB | | Next 100 TB/month | $0.07 per GB | | Greater than 150 TB/month | $0.05 per GB |

On a monthly basis, this hypothetical application requires:

• 5 TB of storage
• 1 million Class A operations
• 100 million Class B operations
• 500 TB of data transfer

The Cloudflare R2 vs Amazon S3 price comparison table for this scenario is:

| S3 Data Transfer Tier | Cloudflare R2 | Amazon S3 | | --- | --- | --- | | Storage | $75 | $115 | | Class A operations (POST) | $4.50 | $5 | | Class B operations (GET) | $36 | $40 | | Data egress | — | $28,800 | | Total | $115.5/mo | $28,960/mo |

Sure, Amazon AWS offerings are quite broad and complex, and there must be a solution that better fits this particular case. However, the Cloudflare R2 plan turned out to be 99.99% cheaper than the S3 Standard plan.

Moving From S3 to Cloudflare R2: Costs and How To Do ItYou might now be convinced to switch from your S3 provider to R2, but what about the egress fees to transfer all your data from your cloud provider to Cloudflare? That could take months and cost you several thousand dollars!

Cloudflare is well aware of the data migration problem and has designed two solutions for that:

• Super Slurper: Move all your data from Amazon S3 or Google Cloud Storage to R2 in a single operation. It preserves custom object metadata from the source bucket by copying it to the migrated objects, and it does not delete any objects from the source bucket. This tool is fast but requires you to pay egress fees from the source provider.
• Sippy: Incrementally migrate from S3 providers to Cloudflare R2 without paying unnecessary egress fees. When an object is requested, it is served from your R2 bucket if it is found. Otherwise, it is simultaneously returned from the source bucket and copied to R2 to minimize data transfer fees. All other operations, including insert and delete operations, continue to work as usual.

Both are free to use and charge only for the Class A operations that the tool needs to perform to transfer data to your R2 bucket.

Advantages and Limitations of R2 Compared to S3Before blindly moving from Amazon S3 to Cloudflare R2, you must analyze in detail the pros and cons of R2 versus S3.

👍 Pros:

• Cheaper than Amazon S3 on data storage and operations
• Much more generous free tier
• No egress/data transfer fees
• S3-compatible API
• Easier setup, with automatic region configuration
• Undiscoverable bucket names
• Simple data migration with Super Slurper and Sippy

👎 Cons:

• Not all S3 API operations are currently supported
• Super Slurper only works with Amazon S3 and GCS, can only transfer objects less than 1 TB, and does not guarantee the same ETags
• More limited ecosystem compared to AWS

ConclusionIn this article, you explored Cloudflare R2 and compared it to Amazon S3. You now know that R2 is a cost-effective cloud storage solution with an S3-compatible API. Thanks to its no-egress fees policy, R2 can save you thousands of dollars a month. However, migrating from Cloudflare R2 to Amazon S3 would mean losing the ecosystem advantages of keeping your data within the AWS infrastructure.

The post Cut Down S3 Storage Costs with Cloudflare R2 appeared first on Semaphore.

View Details

Traditional web hosting services, such as shared hosting, have long been the preferred choice for hosting sites and web applications. However, the web hosting landscape has evolved considerably in recent years. With advancements in technology and the rise of innovative platforms, numerous low-cost alternatives are now available for individuals and businesses seeking reliable hosting solutions.

In this article, you will understand what traditional web hosting services are, what they offer, and how much they cost. You will then explore five modern alternatives that offer affordability without compromising on performance or functionality.

Let’s dive in!

Traditional Web Hosting Services: Types, Cost, and ProvidersA web hosting service is an online service where a provider reserves space on its servers for storing and serving websites. This allows individuals or companies to host sites and make them publicly accessible on the Internet. Usually, these services can also host files, databases, and domain-specific mailboxes.

There are three types of traditional web hosting services:

• Shared
• Dedicated
• Managed

Let’s dig into them, explaining what they offer, analyzing how much they cost, and mentioning some key providers.

Shared Hosting ServicesShared hosting providers share the resources of a single server among many different domains. This makes shared hosting a cost-effective solution for sites with limited traffic, such as blogs or small sites.

• Cost range: A few dollars per month
• Some well-known providers: Bluehost, HostGator, Hostinger, SiteGround, InMotion

Dedicated Hosting ServicesDedicated hosting providers assign one or more servers to each domain exclusively. In other words, servers are not shared with other websites or applications. That means greater performance, security, and control. However, it also implies that server management in on the customer.

Note that many shared hosting providers also offer dedicated hosting plans.

• Cost range: From over $50 per month to hundreds of dollars per month
• Some well-known providers: InMotion, Liquid Web, Bluehost, A2 Hosting, AccuWeb Hosting

Managed Hosting ServicesManaged hosting combines the benefits of dedicated hosting with expert management services. Unlike shared hosting, where multiple websites share the same server, managed hosting provides dedicated servers to individual companies or users. The hosting provider takes care of all server maintenance tasks, including security, software and hardware upgrades, and backups. Managed hosting is ideal for companies that want the reliability and performance of dedicated hosting without needing in-house expertise to maintain the server.

These services usually provide specific solutions such as Managed WordPress, Managed WooCommerce, Managed Magento, and the like.

• Cost range: From over $20 per month to hundreds of dollars per month, depending on the managed technology
• Some well-known providers: WP Engine, Kinsta, Liquid Web, EasyWP, tsoHost

Top 5 Inexpensive Alternatives to Web Hosting ServicesYou know that traditional hosting services range from a few dollars to hundreds of dollars per month. Aren’t there less expensive solutions? Of course, there are!

If you are eager to find out what these solutions are, take a look at the table below:

| Service | Cost | Number of sites | Ideal projects | Supported Sites | Supported Technologies | | --- | --- | --- | --- | --- | --- | | Free Hosting Services | Free | 1-5 | Small sites or non-commercial projects | Static sites and/or web applications | HTML, CSS, PHP, Node.js, React, Next.js, WordPress, and others | | Web Hosting via GitHub Pages | Free | One for each public repository | Non-commercial personal projects or static sites | Static sites and client-side sites | HTML, CSS, JavaScript, static site generators, some client-side technologies | | CDN Hosting | Free—$50+/mo | Potentially unlimited | Static sites | Static resources | Static HTML, CSS, JavaScript, and image files | | Cloud Hosting | <$1/mo—$1000+/mo | Potentially unlimited | Projects with fluctuating traffic | Static and dynamic sites, web applications, and more | Any | | VPS Hosting | $2+/mo—80+/mo | Potentially unlimited, but depends on the resources of the VPS | Projects needing variable resources | Static and dynamic sites, web applications, and more | Any |

Time to explore the five best alternatives to traditional web hosting services to save money!

Free Hosting ServicesFree hosting services offer space on their servers at no cost, typically for small sites and non-commercial projects. These providers usually offer free plans with limited resources, restricted bandwidth, and basic features, supporting only specific stacks like Node.js or traditional web development technologies like HTML, CSS, and PHP.

Some of the providers that offer free hosting services are:

• InfinityFree: Supports sites based on PHP and MySQL.
• 000WebHost: Supports sites based on PHP and MySQL.
• Netlify: Supports web applications and sites built with React, Next.js, WordPress, and other technologies. It offers continuous deployment from GitHub.

💰 Cost range: Free

👍 Pros:

• No financial investment required
• Suitable for small or personal projects
• Great for testing
• Basic features and functionality for getting started
• Ideal for beginners who do not want or cannot afford to pay for hosting

👎 Cons:

• Limited storage, bandwidth, and customer support
• Only a few technology stacks supported
• Advertisements may be displayed on all pages

Web Hosting via GitHub PagesGitHub Pages is a free hosting service to host static sites directly from the files in a GitHub repository. It works with technologies like HTML, CSS, JavaScript, and natively supports Jekyll as a static site generator for blogging. Other static site generators like Hugo, Gatsby and similar can be used as well with some limitations. With some workarounds, it is also possible to deploy some client-side applications, such as React applications. As of this writing, GitHub Pages does not support server-side languages such as PHP, Node.js, Ruby, or Python.

By default, after publishing a site from a repository, it will be available at https://<username>.github.io/<repository> for individual users and https://<organization>.github.io/<repository> for organizations. However, you can also host GitHub Pages on custom domains as explained in the official guide.

Note that GitHub Pages is available for free in public repositories for both individual users and organizations. For more technical information, explore the GitHub Pages documentation.

💰 Cost range: Free for GitHub users

👍 Pros:

• Free hosting with custom domain support
• Built-in integration with GitHub for version control and automatic deploy at each push
• Supports Jekyll natively
• Automatic HTTPS for secure connections
• Configuration in a few clicks

👎 Cons:

• Limited to static content with no server-side processing capabilities
• Requires knowledge of Git
• Limited storage and bandwidth
• Hosting sites from private repositories requires a Pro plan
• Forbidden for commercial use

Content Delivery Network HostingA CDN (Content Delivery Network) is a network of servers distributed globally to strategically cover as many geographical locations as possible. Its purpose is to deliver web content efficiently to users, reducing latency, and speeding up loading times.

Specifically, a CDN caches and delivers resources such as HTML, CSS, JavaScript, and images from the server closest to the user. Thus, CDNs host static sites by distributing their files over a network of servers around the world to guarantee fast and reliable access to them.

These services usually offer pay-as-you-go plans where you pay per GiB of bandwidth and/or monthly plans in which you pay a fixed amount for a given level of service. This makes CDN hosting an inexpensive and efficient option for static sites. Most CDNs complete their offering with anti-bot protection, image optimization, and many other features.

Some popular CDN providers for hosting static sites are Cloudflare, Amazon CloudFront, Fastly, and Akamai.

💰 Cost range: Free, a few cents per GiB of bandwidth, up to $50+ per month, depending on traffic and/or enabled features

👍 Pros:

• Improved latency times
• Reduced resource load times
• Enhanced global reach with servers worldwide
• Scalability to handle traffic spikes
• Extra features like DDoS protection, image optimization, anti-bot solutions, and more

👎 Cons:

• Limited to static content, with no dynamic processing allowed
• Complex DNS setup for beginners
• May become expensive on sites with high traffic

Cloud HostingCloud hosting uses virtual servers in the cloud to host sites and web applications. It pools computing resources from a network of virtual and physical servers, offering high scalability and flexibility.

In most cases, cloud hosting works on a pay-as-you-go basis, which means that you pay only for the resources you use. That is especially beneficial for projects with unpredictable or fluctuating traffic, as there is no need to worry about overprovisioning or underprovisioning of resources.

Compared with traditional web hosting, cloud hosting can be more robust and reliable. This is because when a site hosted on a traditional provider needs more resources, you manually have to buy a new plan to get more computing power. Instead, cloud hosting automatically scales resources up or down for you.

Some popular cloud hosting providers are Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), DigitalOcean, and Vercel.

💰 Cost range: From a few dollars a month to thousands of dollars a month, depending on usage

👍 Pros:

• Automatic scalability to handle varying traffic levels
• Pay-as-you-go pricing model for cost efficiency, especially on sites with fluctuating traffic
• Redundancy and reliability through distributed infrastructure
• Integration with CI/CD pipelines
• Tons of other useful features and services

👎 Cons:

• Requires technical expertise for setup and management
• Traffic spikes can lead to unexpected costs
• Vendor lock-in

Virtual Private Server HostingA VPS (Virtual Private Server) is a virtualized server environment created by splitting a physical server into multiple virtual servers. VPS hosting providers generally provide access to Linux or Windows virtual servers, with users who are tasked with configuring them from scratch for website hosting.

This hosting experience is similar to that offered by a traditional managed hosting service. The main difference is that VPSs are more flexible in that resources can be added or removed dynamically according to your needs, as they are virtual servers and not physical ones. Moreover, you typically pay based on the resources selected. Thus, in the long run, dedicated VPs can be a more cost-effective solution than managed hosting services.

Popular VPS providers include DigitalOcean, IONOS, InterServer, Vultr, and Linode.

💰 Cost range: From a few dollars per month to dozens of dollars per month, depending on how large the VPS is

👍 Pros:

• Improved performance compared to shared hosting
• Resource scalability
• Root access for full control over server configuration
• A single VPS can host several sites
• Usually cheaper than managed hosting

👎 Cons:

• Requires technical expertise for server management
• Responsibility for security and updates
• Maintenance and support might come at an additional cost
• Security concerns as they are virtual servers

ConclusionIn this article, you learned what types of traditional web hosting services are and how much they cost. Fortunately, there are other low-cost modern solutions. Here, you dug into the five best alternatives to traditional web hosting solutions to save money without compromising on functionality. If you are an individual developer or small business, knowing these low-cost hosting solutions can save you a lot of money!

The post 5 Low-Cost Web Hosting Alternatives appeared first on Semaphore.

View Details

The rise of microservices, cloud platforms, containerization, and distributed systems has made computer systems incredibly powerful, but also increasingly intricate. Ensuring high availability in these complex environments requires innovative solutions. Enter Coroot, an open-source observability tool designed to simplify monitoring and prevent chaos in production. Coroot co-founders Peter Zaitsev and Nikolay Sivko shed light on their vision: empowering engineers to manage this complexity, maintain system uptime, and swiftly resolve issues.

Edited transcriptionWhat is Coroot?“We have a lot of open source and commercial tools that can help you in gathering telemetry data, building your dashboards, alerts, and so on, and we decided to look at the problem from different angles,” says Nikolay Sivko, CEO and co-founder of Coroot, an open-source observability tool. As systems are now more complex than ever (consisting of “microservices, architectures, hundreds or even thousands of services”), Coroot “aims to help engineers to manage this chaos in the production environment,” Nikolay explains.

To this end, Coroot is meant to provide not a cascade of insight, but just the right amount to help solve problems without overwhelming the user. As such, Coroot focuses on delivering actionable answers to user questions, providing information on what’s happening within the system, including errors and their root causes. In other words, Coroot’s bet is on actionability rather than comprehensiveness. “Being able to solve 95% of the problem is much better than having information to solve 100% of the problem, but being overwhelmed and not being able to do that,” says Coroot Co-founder Peter Zaitsev.

“The different approach here is working not with telemetry data directly, but with a model of a system,” says Nikolay. Coroot relies on fine-grained metrics, logs, and traces to build a comprehensive system model that takes into account all components working together, including containers, underlying nodes, network communication, and latency. “Analyzing such a model allows us to provide you with insights,” says Nikolay.

The Power of eBPFA core strength of Coroot is its ease of deployment, compared with other observability tools that require a complex installation and configuration for each specific use case. This is possible thanks to eBPF (Extended Berkeley Packet Filter). As Nikolay explains, “eBPF allows you to inject your small programs into any particular place of the kernel so we can instrument network connections, capture application-level queries, requests, and so on.”

eBPF works without code changes in the kernel itself. Its programs typically store captured data in a limited memory space (like a ring buffer) within the kernel. A separate user-space program then retrieves and processes this data, ensuring that even if the user-space program can’t keep up, the kernel buffer won’t overflow and crash the system.

What’s more, eBPF collects data with minimal performance impact, allowing Coroot to gather essential data with minimal setup, providing initial valuable insights “out of the box.” eBPF programs are verified before execution, guaranteeing they cannot introduce problems like infinite loops that could crash the system.

Embracing OpenTelemetrySimultaneously, Coroot uses OpenTelemetry, a vendor-neutral framework that allows users to avoid vendor lock-in with telemetry tools. OpenTelemetry simplifies collecting telemetry data (metrics, logs, traces) and sending it to various systems without code changes. According to Nikolay, implementing OpenTelemetry “is a good shift on the market because you can easily replicate your telemetry data flow into many systems and you don’t need to change your code anymore; once instrumented, it can be supported by many vendors, including Coroot.”

Failure-Driven Development (FDD)Coroot uses a reactive development approach, where encountered failures inform future development: “We use failure-driven development because we love to reproduce failures and then build a product that can detect it,” says Nikolay. “We produce various scenarios of failure and build products that highlight the most time-consuming piece of your code and the root cause of errors from distributed cases,“ he adds.

On the other hand, the development team has no specific emphasis on unit tests. Unit tests might be written situationally. Due to the complex interactions between different Coroot components (node agent, exporters, core engine), integration testing is the primary focus.

Community and development“Now the majority of code is written by the team because we still are implementing our vision,” says Nikolay. This vision is still evolving but prioritizes features that directly address user needs. In this regard, Coroot plans to expand its deep-dive capabilities beyond Postgres, including highlighting resource-intensive queries in other databases and queue systems.

Still, while the core development is internal, Coroot welcomes contributions for smaller features like webhooks, plugins for specific databases, or integrations with other cloud providers. As for future features, Nikolay explains, “We are focusing now on simplifying the installation process because we want to provide users with a solution that can suggest to you what else you need to instrument.”

The bottom lineTrying Coroot is free and requires minimal setup. There’s a SaaS version available for those who prefer a cloud-based option and the possibility of self-hosting for those comfortable with Kubernetes. For a quick and easy setup, Nikolay and Peter recommend installing Coroot directly within your Kubernetes cluster using Helm.

The initial Coroot setup provides immediate value by offering insights into your entire system, including control plane components and all services, without additional configuration.

Additionally, Coroot welcomes feedback to refine its product vision. Users can engage through various channels:

• Coroot website
• GitHub repository
• Slack community

While Coroot initially targeted Kubernetes environments, it has broadened its reach. The goal is to support various infrastructure components, including legacy databases, virtual machines, and managed services like AWS RDS and Elasticache: “We aim to support all the components to provide users with 100% coverage of the infrastructure because it’s super important to be able to see the big picture,” says Nikolay.

The post Peter Zaitsev and Nikolay Sivko on simplifying observability appeared first on Semaphore.

View Details

As discussed in the previous article, Docker secrets are a secure way to manage sensitive information in Dockerized environments. You got to see how you could incorporate secrets into compose files and also how you could manage secrets using the docker secret command.

As a follow-up to the previous article, we’ll be looking at three practical use cases of Docker secrets. We’ll go through how to use Docker secrets in a Dockerfile, manage WordPress secrets with Docker secrets, and roll back a secret. Through these practical examples, you will better understand how you can use Docker secrets in real-world scenarios.

PrerequisitesTo fully understand the hands-on examples in this article, you should have a basic understanding of Docker and Docker secrets. If you’re new to Docker secrets, you can check out the previous article to get started.

Additionally, you should have Docker installed on your machine preferably with Docker Swarm enabled.

Using Docker Secrets in a DockerfileIn some cases, you might need to incorporate secrets directly into your Docker image. This could be necessary when you need to build an image that requires a secret to be present at build time. In this example, we’ll look at incorporating a secret into a Docker image using a Dockerfile.

Step 1: Create a SecretFirst, you must create a secret that will be incorporated into the Docker image. You can create a secret using the docker secret createcommand.

For this tutorial, let’s say you have an AWS access key stored in a file named aws_credentials in your home directory ($HOME/.aws/credentials).

Step 2: Create a DockerfileCreate a Dockerfile in your project directory. This Dockerfile will define the steps needed to build your Docker image, including how to access and use the secret.

FROM your_base_image# Create a directory for the secretRUN mkdir -p /run/secrets# Copy the secret file into the containerCOPY $HOME/.aws/credentials /run/secrets/aws_credentials# Use the secret in your build processRUN --mount=type=secret,id=mytoken \ TOKEN=$(cat /run/secrets/aws_credentials) \ # Your build commands here Replace your_base_image with the base image you’re using for your Docker image.

This Dockerfile creates a directory for the secret, copies the secret file into the container, and then uses the secret in the build process. The --mount=type=secret,id=mytoken flag allows you to pass the secret to the build process securely.

In an actual scenario, you would replace the # Your build commands here with the specific commands needed to build your Docker image.

NOTE: As emphasized in the previous article, sharing secrets over version control systems like Git is not recommended. For this example, we’re assuming the secret file is stored locally on your machine, so it’s not shared with others.

Step 3: Build Your Docker ImageNow, build your Docker image using the docker build command. Make sure to use the --secret flag to pass the secret to the build process.

docker build --secret id=mytoken . This command will build your Docker image while securely passing the secret to the build process.

Step 4: Use the Secret in Your ApplicationYou can now use the secret in your application as needed. For example, if you’re using the AWS access key in your application to interact with AWS services, you can access it using the TOKEN environment variable set in the Dockerfile.

NOTE: Remember to replace mytoken with a meaningful identifier for your secret. Also, ensure that the secret file (aws_credentials in this example) is stored securely and is only accessible to authorized users.

Managing WordPress SecretsWordPress is a popular content management system that often requires sensitive information like database passwords to be securely managed.

In this example, we’ll create single-node MySQL and WordPress services using Docker secrets to manage sensitive credentials. We’ll set up a MySQL database for WordPress and configure WordPress to connect to MySQL.

OverviewThis tutorial can get a bit complex, so let’s break it down into steps:

1. Generate random alphanumeric passwords for MySQL root and WordPress database.
2. Create Docker secrets for MySQL and WordPress.
3. Set up a MySQL service with Docker secrets.
4. Configure a WordPress service to use MySQL with Docker secrets.
5. Verify the services are running and test WordPress functionality.

Step 1: Generate Docker SecretsGenerate random passwords for MySQL root and WordPress database:

```

Generate MySQL root passwordopenssl rand -base64 20 | docker secret create mysql_root_password -# Generate WordPress database passwordopenssl rand -base64 20 | docker secret create mysql_password -

``` You should now have two Docker secrets with an output similar to:

uxm1a4sjjsg7enq2m1dtmvnon5judi7jm6d02xs99lj8061wul Step 2: Create MySQL ServiceCreate a MySQL service with Docker secrets:

docker network create -d overlay mysql_privatedocker service create \ --name mysql \ --replicas 1 \ --network mysql_private \ --mount type=volume,source=mydata,destination=/var/lib/mysql \ --secret source=mysql_root_password,target=mysql_root_password \ --secret source=mysql_password,target=mysql_password \ -e MYSQL_ROOT_PASSWORD_FILE="/run/secrets/mysql_root_password" \ -e MYSQL_PASSWORD_FILE="/run/secrets/mysql_password" \ -e MYSQL_USER="wordpress" \ -e MYSQL_DATABASE="wordpress" \ mysql:latest Your output should look similar to:

v87xhur2iqhwhvmql11345l7koverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged This command creates a MySQL service with the specified secrets and environment variables. The MySQL service will be connected to the mysql_private overlay network.

Step 3: Create WordPress ServiceCreate a WordPress service that connects to MySQL using Docker secrets:

docker service create \ --name wordpress \ --replicas 1 \ --network mysql_private \ --publish published=30000,target=80 \ --mount type=volume,source=wpdata,destination=/var/www/html \ --secret source=mysql_password,target=wp_db_password \ -e WORDPRESS_DB_USER="wordpress" \ -e WORDPRESS_DB_PASSWORD_FILE="/run/secrets/wp_db_password" \ -e WORDPRESS_DB_HOST="mysql:3306" \ -e WORDPRESS_DB_NAME="wordpress" \ wordpress:latest You should see output similar to:

whoumrpbcry3exply4fl9euueoverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged This command creates a WordPress service that connects to the MySQL service using Docker secrets. The WordPress service will be accessible on port 30000.

Step 4: Verify and TestFinally, verify that the services are running and test WordPress functionality:

```

Verify servicesdocker service ls# Test WordPressdocker service ps wordpress

``` You output should look similar to:

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTSq2qzd1d2hup1 wordpress.1 wordpress:latest docker-desktop Running Running 2 minutes ago Access http://localhost:30000/ from any swarm node to set up WordPress using the web-based wizard. Verify that WordPress is operational and its state is saved across service restarts.

Figure 1:* WordPress Installation PageNOTE:* You need an active Docker Swarm to run the services. If you’re using Docker Desktop, you can enable Swarm mode by running docker swarm init.

Rotating Back a SecretRotating a secret means changing its value without affecting the services that use it. This is an important security practice to prevent unauthorized access to sensitive information.

This tutorial will demonstrate how to rotate a secret in Docker, building upon the previous use case. We’ll create a new secret with a new MySQL password, update the MySQL and WordPress services to use it, and then remove the old secret.

Step 1: Create a New SecretGenerate a new password and store it as a secret named mysql_password_v2:

openssl rand -base64 20 | docker secret create mysql_password_v2 -vwqvnkm1gploicqk2g32vwpf2 Step 2: Update ServicesUpdate the MySQL service to grant it access to the old and new secrets. Remove the old secret and add the new one:

docker service update \ --secret-rm mysql_password mysqldocker service update \ --secret-add source=mysql_password,target=old_mysql_password \ --secret-add source=mysql_password_v2,target=mysql_password \ mysql You should see output similar to:

mysqloverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Waiting 5 seconds to verify that tasks are stable... service update paused: update paused due to failure or early termination of task f98yfogvf606ddrdc0z3b1pafmysqloverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged This command updates the MySQL service to use the new secret while keeping the old secret for backward compatibility.

Step 3: Change MySQL PasswordChange the MySQL password for the WordPress user using the mysqladmin CLI. This command reads the old and new passwords from the secret files:

docker container exec $(docker ps --filter name=mysql -q) \ bash -c 'mysqladmin --user=wordpress --password="$(< /run/secrets/old_mysql_password)" password "$(< /run/secrets/mysql_password)"' You should see output similar to:

mysqladmin: [Warning] Using a password on the command line interface can be insecure.Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety. This command changes the MySQL password for the WordPress user to the new password.

Step 4: Update WordPress ServiceUpdate the WordPress service to use the new password. Trigger a rolling restart of the service:

docker service update \ --secret-rm mysql_password \ --secret-add source=mysql_password_v2,target=wp_db_password \ wordpress

You should see output similar to:bashwordpressoverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged This command updates the WordPress service to use the new secret and removes the old secret. Step 5: Verify FunctionalityOn the WordPress site, verify that you can still access and interact with the site as expected. Ensure that the MySQL password change did not affect the WordPress functionality.

NOTE: You might be wondering how all this can work in CI/CD pipelines such as Semaphore. To make this possible, you’ll need scripts. So, instead of manually running these commands on your machine, the CI/CD pipeline will execute this script and commands automatically whenever there is a new code push or a trigger event. To try it out, check out this GitHub repository that goes through this process.

However, Semaphore has a much simpler way of including secrets into your project. You can check this page on creating and managing secrets to know more.

Step 6: Revoke Access and CleanupAfter verifying that the services are operational, revoke access to the old secret and clean up the old secret:

docker service update \ --secret-rm mysql_password \ mysqldocker secret rm mysql_password

You should see output similar to:bashmysqloverall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged mysql_password ConclusionIn this article, we looked at three practical use cases of Docker secrets. We explored how to incorporate secrets into a Docker image using a Dockerfile, manage WordPress secrets using Docker secrets, and rotate a secret in Docker.

With these examples, using secrets in various scenarios becomes more straightforward. You can now apply Docker secrets to your projects to securely manage sensitive information and improve the security of your Dockerized applications.

The post Three Practical Use Cases of Docker Secrets appeared first on Semaphore.

View Details

According to Gartner estimates, by 2027 more than 90% of global organizations will be running containerized applications in production. A significant increase from fewer than 40% in 2021. As containerization gains traction and becomes the de facto standard for deploying applications, Kubernetes is seeing a sharp increase in usage across enterprises.

Having said that, a recent survey conducted by CNCF in partnership with FinOps foundation revealed an increase in Kubernetes cloud cost across the board during 2023. A surprising finding is that most respondents needed help tracking and predicting Kubernetes cost-effectively. A wise man once said, “A penny saved is a penny earned,” but measuring costs correctly in Kubernetes environments is complex.

This article discusses best practices and tools for implementing cost optimization and gaining accurate visibility for your Kubernetes workloads.

Challenges in Managing Kubernetes Cloud CostLet’s quickly take a look at some of the challenges of managing costs with Kubernetes:

• Kubernetes workloads are dynamic, making it difficult to accurately calculate the cost of the resources used by your apps.
• We frequently deploy applications to a multi-tenant Kubernetes cloud cluster, which complicates accurately estimating resource costs. While cloud vendors make it relatively easy to know that your Kubernetes cluster is costing you $100 per day, it can be difficult to estimate how many resources, i.e. compute and memory, a team or project is using.
• Even if everything is reduced to CPU, memory, and load balancers, various external considerations such as shared resources and cluster management fees can complicate your Kubernetes cluster cost estimation.
• Overprovisioning of resources and a lack of awareness and responsibility among individuals and teams can increase the cost estimate of your Kubernetes cluster cloud. Overprovisioning can also lead to the tragedy of commons situation where there is no incentive to be modest while dimensioning deployments.
• In the absence of standardized processes and tools to visualize Kubernetes cost, budgeting or spending analysis becomes difficult.
• Sometimes, when you are unsure about your workload consumption trends, which fluctuate often, it is hard to get accurate information on Kubernetes cloud cost.

Splitting Down CostsKnowing how much your managed Kubernetes cloud cluster will cost ahead of time is preferable to being surprised when you receive your monthly bills. The following components make up the typical cost of running an application on a managed Kubernetes cloud cluster:

• CPU and memory
• Persistent storage
• Ingress/Egress
• Network
• Common services (i.e., control plane and service mesh)
• Cluster management fee

How Cloud Providers ChargeYour organization’s decision to harness the power of Kubernetes depends on a set of parameters, and cost is crucial. But do you know how different cloud providers charge you for managed Kubernetes instances?

Every cloud provider charges you differently for the managed control plane. For example, Digital Ocean comes with a free control plane. This free service includes management for processes like etcd, kube-apiserver, kube-controller-manager, kube-scheduler, cloud-controller-manager, and other services. Civo is another managed Kubernetes service vendor that offers the control plane for free

While on the other hand, Google Kubernetes Engine and AWS Elastic Kubernetes Service charge for the control plane usage. Keeping track of all factors impacting Kubernetes cluster cost can be overwhelming.

Strategies to Manage Kubernetes Cloud CostsThe following are some best practices teams can follow to optimize their Kubernetes cloud costs.

• Right-sizing your Kubernetes cluster nodes is crucial in optimizing resource utilization. If your containers are too large and consume most of your node resources, adding more containers on that node would be a task.
• You should right-size pods using Kubernetes resource requests and limits by specifying the minimum and maximum amount of resources your application can request.
• Leveraging autoscaling and downscaling features of Kubernetes using the horizontal pod autoscaler and cluster autoscalercan drastically reduce your cloud expenditures.
• Kubernetes resource bin packing is another technique you should consider for cost-efficient utilization of your infrastructure. In this technique, multiple containers are consolidated into fewer nodes to increase the utilization of your cluster nodes.
• Cloud discounts (i.e., AWS saving plans)
• Startup programs (i.e., Y Combinator companies get $100k in GCP credits)
• If your Kubernetes workloads are stateless, you can run them on spot instances to save cost.
• Negotiate and ask for volume-based discounts

Tools to Manage Kubernetes CostsAscertaining and managing your Kubernetes costs is a complex task, and you need multiple tools in your arsenal to do some of the heavy lifting.

Cast AICAST AI is a Kubernetes automation platform that claims to cut cloud costs by half.

• CAST AI can scan your Kubernetes cluster within a few minutes and give you a cost-saving report to know how much you could have saved using its features. Furthermore, if you agree with the report’s findings, you can structure CAST AI to implement those changes to cut costs immediately and automatically.
• CAST AI also has an AI-driven instance selection mechanism that, based on your requirements, decides on the best instance type at the minimum cost.
• CAST AI has bin packing features that allow you to compact pods in fewer nodes and later on delete nodes when they no longer have workloads scheduled on them. In this approach, your nodes reach high utilization, and you don’t have to pay for resources that aren’t utilized.

Kube-downscalerkube-downscaler is an open-source framework that can reduce your Kubernetes resources to zero when they are not being used actively.

• kube-downscaler scales down Kubernetes resources during non-working hours. It’s like Thanos snapping his fingers, and all your Kubernetes pods are effectively scaled down to zero.
• You can define schedules based on your usage pattern, and then it will scale down your cluster. For example, you can scale down the cluster on weekends.
• It is great for your lower environments, where you might not need the infrastructure to be available 24/7.

HarnessHarness is a modern software delivery platform powered by AI. It has built-in capabilities to manage your cloud bills with its Cloud Cost Management feature.

• Harness provides in-depth, actionable insights into your dynamic idle Kubernetes resources by effectively shutting down non-production idle resources with its cloud AutoStopping feature. Harness claims that by using this feature, teams can save up to 70% on non-production cloud costs, all with a simple one-time setup that doesn’t need maintenance.
• Estimate how much of your existing cloud costs you can save using its savings calculator.
• Detecting and flagging anomalies in your cloud consumptions by analyzing historical spending patterns.

KubecostKubecost is the most popular open-source tool for getting a real-time bird’s eye view of Kubernetes costs. It uses preconfigured Grafana dashboards with its opinionated setup (tailored to your needs) to monitor Kubernetes cluster costs. You can install it using a Helm chart.

• It is a unified Kubernetes cost-monitoring solution that supports all major cloud providers.
• Automatically generates Kubernetes cost optimization recommendations via its UI or API. These recommendations can be dynamically applied for immediate results.
• Provides real-time alerting capabilities so that teams can take control of their Kubernetes-related costs and stay within budgeted limits.
• Breaks down the cost allocation to Kubernetes resources such as deployments, namespaces, labels, pods, and containers (but not limited to these). You can transparently and accurately share the cost among your teams and tally it with your cloud bills.

OpenCostOpenCost is an open-source cost monitoring tool for cloud-native environments. It is a CNCF sandbox project created by Kubecost.

• If you have budget constraints, this is the only tool built by Kubernetes experts and practitioners that is freely available in the market.
• It relies on vendor-neutral OpenCost Specification to measure and allocate Kubernetes clusters’ infrastructure and container costs.
• OpenCost API enables real-time and historical reporting on Kubernetes cloud costs based on on-demand list pricing.

LoftLoft’s Kubernetes platform has excellent features to help you manage resource costs.

• Sleep mode can scale down your Kubernetes environment by putting namespaces to sleep and purging all pods inside a namespace during periods of inactivity. Kubernetes resource configuration is not purged and remains in the cluster.
• Auto-delete, as the name implies, deletes namespaces and virtual clusters that have been idle, old, or unused for some time.

Harness Case StudyRelativity, a SaaS data company, reduced its cloud costs by millions in five months using Harness cloud cost management features. Relativity doubled the density of Kubernetes pods per node from 40 to 100 by analyzing the poor utilization of Kubernetes pods operating on cloud infrastructure (nodes), and in some large nodes, 100 pods per node. The financial impact of this was a 40% reduction in Kubernetes costs per day. They saved one full-time employee from the expenditures of developing an in-house cost management platform for product and engineering teams using Harness Cloud Cost Management.

Final ThoughtsWhile it is customary for engineering teams to overprovision resources to run their Kubernetes clusters, this approach may not be sustainable—someone must pay in the end. So, whether you are a seasoned expert or just starting with Kubernetes, you must consider the costs of maintaining a Kubernetes cluster.

The post Optimizing Kubernetes Cloud Costs appeared first on Semaphore.

View Details

UI rendering and animations are what give an app a great and interactive user experience. As interesting as this concept is, so is how frustrating it can become if things go wrong. And one way that could happen is tests being flaky.

There are actually multiple reasons for flaky tests in rendering or animations, but one of the main reasons is timing.

For instance, you might have a spring animation of a button displaying. Now testing this might result in flakiness, all because the animation finished some milliseconds too late due to a random background running process.

In this guide, we’ll be using Jest and React testing libraries to describe real-world examples of tests for better understanding. However, you can follow along, irrespective of your stack.

Root causesFlaky tests, in general, have quite a number of common causes. However, when being specific to UI rendering and animations, here are the main root causes:

Timing IssuesAnimations by default often come with setTimeout delays or transition-duration properties. These timings are sometimes what make assertions fail resulting in flakiness.

Take a look at this test for a component with an animation whereby its width gets increased with the click of a button:

test("Box width increases", () => { render(<Box />); const box = screen.getByTestId("box"); expect(box).toHaveStyle({ width: "70px" }); fireEvent.click(screen.getByRole("button")); expect(box).toHaveStyle({ width: "100px" });}); The problem right here is that after clicking the button to increase the width, the assertion might execute before the animation of the width increase finishes depending on any timing events.

Thus, the main challenge in rendering and animation testing comes down to making sure that assertions are made after the animation has fully executed and not earlier. A nice approach would be to wait for the animation to complete; in React testing, this can be achieved with the use of waitFor.

Environmental DependenciesThe system resources come into play in this case. In a way, tests created for rendering complex animations can be dependent on system resources like CPU or memory.

Now this won’t be an issue for machines that are capable. However, if the machine has limited resources, animation rendering could be slow and less smooth, thus triggering random test failures.

Another instance of environmental dependencies can be network connectivity. Let’s say you have an animation that relies on fetching external assets; this alone introduces unexpected factors as network fluctuations can influence the animation behavior. A way to fix this would be with the use of mocks.

Libraries DependenciesOften times, animations used are from libraries, as it is much faster and more efficient than writing the code from scratch.

In React for instance, there are tons of animation libraries like Framer Motion, React Spring, etc. However, these libraries could cause flaky tests when:

• the version of the third-party library and the current version of the technology stack used are not compatible. This case is sometimes silent, as errors might not be displayed, just that the tests results seems to be unstable.
• the animation libraries used have dependencies that interfere with one another.
• the animation libraries have internal bugs that haven’t been fixed by the library developers which only show up during testing.
• developers use animation libraries in a way it wasn’t designed for. Interestingly enough, the library might perform the task as expected but since it was used incorrectly, testing can provide inconsistent results.

Fixing Flaky Tests in Rendering and AnimationNow, to make this guide more elaborate and easier to understand, we’ll provide four real-world code examples and show the wrong way causing flaky tests and the right way to fix them.

Rendering of a Newsletter PromptIn this example, this component renders a simple form with an email input and a subscribe button. When submitted, it logs the email and resets the state.

function NewsletterPrompt() { const [email, setEmail] = useState(""); const [subscribed, setSubscribed] = useState(false); const handleSubmit = (e) => { e.preventDefault(); setSubscribed(true); }; return ( <div> {!subscribed ? ( <div> <h2>Subscribe to Newsletter</h2> <input type="email" placeholder="Email address..." value={email} onChange={(e) => setEmail(e.target.value)} /> <button type="submit" onClick={handleSubmit}> Subscribe </button> </div> ) : ( <p>Subscribed successfully</p> )} </div> );} Now let’s write a test for the component that’ll show how flakiness can happen:

test("Newsletter prompt renders and can be subscribed to", () => { render(<NewsletterPrompt />); const emailInput = screen.getByPlaceholderText("Email address..."); const subscribeButton = screen.getByText("Subscribe"); fireEvent.change(emailInput, { target: { value: "xyz@example.com" } }); fireEvent.click(subscribeButton); expect(screen.getByText("Subscribed successfully")).toBeInTheDocument();}); The problem here is that making assertions directly like this could run before the component even re-renders to show updates of the submitted email.

This test is bound to be flaky because it relies on the component being rendered synchronously, which might not always be the case. As rendering can sometimes be asynchronous.

A fix for this test would be something like this:

test("Newsletter prompt renders and can be subscribed to", async () => { render(<NewsletterPrompt />); const emailInput = screen.getByPlaceholderText("Email address..."); const subscribeButton = screen.getByText("Subscribe"); fireEvent.change(emailInput, { target: { value: "xyz@example.com" } }); fireEvent.click(subscribeButton); expect( await screen.findByText("Subscribed successfully"), ).toBeInTheDocument();}); In this fix, we used async/await and the screen.findByText method to expect and wait for the text content element to show up in the DOM, which would prove that the component has indeed re-rendered with the “Subscribed successfully” message.

An Animated ButtonLet’s say you have a button that animates its color from orange to blue when clicked and reverts to orange when clicked again.

export default function AnimatedButton() { const [isAnimated, setIsAnimated] = useState(false); const handleClick = () => { setIsAnimated(!isAnimated); }; const buttonStyle = { width: "100px", height: "50px", backgroundColor: isAnimated ? "blue" : "orange", transition: "backgroundColor 0.5s ease", }; return ( <button style={buttonStyle} onClick={handleClick}> Animate </button> );} Now here’s a common but incorrect way to write the test for this button color change, as it could show flaky behavior:

import { render, fireEvent } from "@testing-library/react";import AnimatedButton from "./AnimatedButton";test("background color changes", () => { render(<AnimatedButton />); const button = screen.getByRole("button", { name: "Animate" }); fireEvent.click(button); expect(button).toHaveStyle({ backgroundColor: "blue" }); fireEvent.click(button); expect(button).toHaveStyle({ backgroundColor: "orange" });}); This test looks really straightforward; it renders the button, clicks it, and then asserts the expected color changes. However, there’s a bit of a weak spot, which is where flakiness comes in.

The reason this test might sometimes show flaky results is that after each click event, it assumes that the animation completes immediately, as we aren’t considering the timing of the animation before asserting the state of the button’s background color.

Instead, this would be the right way:

test("background color changes", async () => { render(<AnimatedButton />); const button = screen.getByRole("button", { name: "Animate" }); fireEvent.click(button); await waitFor(() => expect(button).toHaveStyle({ backgroundColor: "blue" })); fireEvent.click(button); await waitFor(() => expect(button).toHaveStyle({ backgroundColor: "orange" }), );}); With this simple change of using waitFor, we can be certain that the test would allow the animation to be completed before asserting the background color of the button. This makes sure that the test assertion is in sync with the UI’s rendering update.

Rendering of a Basic ModalFor this example, let’s use a simple modal component that when a button is clicked, the visibility of the modal changes:

function AnimatedModal() { const [isVisible, setIsVisible] = useState(false); const toggleVisibility = () => setIsVisible(!isVisible); return ( <div> <button onClick={toggleVisibility}>Toggle Modal</button> {isVisible && <div data-testid="animated-modal">Animated Modal</div>} </div> );} Now as simple as this component is, testing if it renders correctly can become a hassle because of the animation.

Here is a wrong way to write the test:

test("Toggling animated modal", () => { render(<AnimatedModal />); const toggleButton = screen.getByRole("button", { name: "Toggle Modal" }); expect(screen.queryByTestId("animated-modal")).not.toBeInTheDocument(); fireEvent.click(toggleButton); expect(screen.getByTestId("animated-modal")).toBeInTheDocument();}); As we explained in the previous example, this test also asserts the presence of the modal content immediately after the click event. So due to the animations being asynchronous, this test is bound to be a flaky one.

Here’s the right way:

test("Toggling animated modal", async () => { render(<AnimatedModal />); const toggleButton = screen.getByRole("button", { name: "Toggle Modal" }); expect(screen.queryByTestId("animated-modal")).not.toBeInTheDocument(); fireEvent.click(toggleButton); await waitFor(() => { expect(screen.queryByTestId("animated-modal")).toBeVisible(); }); fireEvent.click(toggleButton); await waitFor(() => { expect(screen.queryByTestId("animated-modal")).not.toBeInTheDocument(); });}); In this fix, we used waitFor to assert the modal’s presence or absence visibility only after the animation has been completed. This gives us the certainty that the test is more reliable.

Strategies to Reduce Flaky Tests in Rendering and AnimationSo when writing tests specifically for UI rendering and animations, here are some things to look out for or what you should use instead to reduce the chance of flaky tests showing up:

• Always review the test logic written as it is one of the common mistakes made as explained in the section above.
• Network delays or fluctuations should be accounted for and mock functions can come in handy in fixing this.
• When testing on UI rendering, avoid using DOM selectors like .querySelector() to get elements, instead use the queries provided by the testing library like getBy, queryBy, or findBy.
• Ensure that the version of the animation library used is compatible with the version of the tech stack used.
• Keep the rendering or animation tests in smaller units i.e. try to only test one component at a time. This is more efficient and easier to debug flakiness.
• Know when to use the appropriate assertion matchers, for instance when testing animations, toBeVisible() is better suited than toBeInTheDocument().
• Use waitFor for assertions and not for firing events.
• You can use jest.useFakeTimers() as an alternative to waitFor for further control of testing.
• Some developers skip/disable animations when testing, while it’s not recommended, it is an option.

ConclusionFlaky tests in rendering and animations can actually be difficult to deal with but with a proper understanding of the common causes specifically timing complications, the flakiness of tests can be reduced tremendously.

For further reading on flaky tests, check out these resources:

• Flaky Tests In React: Detection, Prevention and Tools
• Best Practices for Identifying and Mitigating Flaky Tests
• How to Avoid Flaky Tests in Playwright

The post How To Fix Flaky Tests in Rendering and Animation Workflows appeared first on Semaphore.

View Details

In software development, an application is only as good as its documentation. Good technical documentation requires a lot of work but is one of the best investments a company can make.

To keep up with the need for quick and accurate guides, companies, today, are using AI tools for help. So, in this article, we’ll take a look at three AI documentation generators, check out what they offer, and weigh their pros and cons.

Three Tools for Automatic Software DocumentationIn this section, we’ll explore the capabilities of three prominent AI-driven documentation generators:

• Scribe
• DocuWriter
• Bito AI documentation generator

We’ll explore their features and make a comparison among them to enlighten similarities, differences, and use cases.

ScribeScribe is an open source, AI-powered documentation generator.

This tool turns any process into step-by-step tutorials along with text, code snippets, internal links, and screenshots. So, the ideal usage of this tool is to create processes on the fly, as you go.

For example: do you know that Excel formula you use barely every day but it’s so complicated you never remember it? Well, here’s where Scribe comes into handy: while you make the process, Scribe creates the documentation for you. Once and for all.

No more screenshots, GIFs, or videos scattered here and there: Scribe creates the documentation as you make the process, making it consistent.

Check out how easy is to create (and share!) documentation with Scribe here.

DocuWriterDocuWriter it’s an AI-powered documentation generator that creates code and API documentation from your source code files.

In particular, DocuWriter uses AI to generate the documentation for your code, allowing you to save time while having consistent documentation, updated with the latest changes.

The power of DocuWriter not only relies on automatically creating the documentation. It also creates tests and code refactors from your source code, allowing you to concentrate only on what matters: solving problems through your favorite programming language.

Last but not least, you can use it directly in VS CODE, thanks to the dedicated integration.

Bito AI Documentation GeneratorBito documentation generator is an AI-automated software documentation generator that ensures the documentation is always up-to-date with the latest code changes.

By Providing Bito a directory, it will automatically generate a detailed overview, visualization, and documentation for each file including:

• Summary of the file
• Dependencies
• Documentation regarding classes, modules, functions, and methods.

If the execution of Bito is successful, the output provided creates a directory that includes:

• Modules documentation. These are individual markdown files for each module, that explain the module’s purpose, functions, and interactions.
• Aggregated documentation. This is a comprehensive markdown file that consolidates the documentation from each module. This file also includes flow maps and visuals.

A Comparison Among ThemAs not all the tools are useful for every scenario, in this section we want to make a comparison among these three documentation generators to understand some use cases they might be better suitable for.

Scribe is particularly useful to document processes and procedures. Thanks to how it is developed, in fact, it registers the movements the mouse is making on the screen during the process and it creates a step-by-step guide out of it.

This comes in handy in different situations like:

• Troubleshooting.
• Processes and procedures, both internal and external to the company you’re working for.

So, Scribe may be useful for non-developer users who need to create process documentation but want to use software that speeds the process up, avoiding the creation of images and videos for, then, creating the structure elsewhere (like in Microsoft Word or Confluence, as it typically happens).

DocuWriter, on the other hand, is a documentation generator specifically suited for a developer audience, helping it automatically create the documentation for their APIs.

As there are a lot of documentation generators that create API documentation, DocuWriter emerges for the following features:

• It creates tests and code refactors from the code files, not only the documentation.
• It can be integrated into VS CODE.
• It provides documentation both in markdown and PDF.
• It creates the documentation for your preferred programming language, without any exceptions.

Even Bito is particularly suited for a developer audience. The main difference between Bito and DocuWriter is that DocuWriter created the documentation for APIs while Bito created the documentation related to the source code of your project providing:

• Documentation for modules and functions and their interactions.
• Visuals and flow maps to represent modules’ interactions.
• The possibility to ignore directories during the process of creating the documentation.

Pros and Cons of Using AI Documentation ToolsAs AI is dominating almost every scenario and industry today – at least, in IT – it doesn’t mean that an AI tool is the perfect solution for your needs.

In fact, documentation generators aren’t always the right solution to your documentation problems. So, here we want to explore the pros and cons of using AI documentation generators.

Pros:

Among the pros, we can consider the following:

• Efficiency. AI documentation generators can automatically generate comprehensive documents much faster than manual writing, saving significant time and resources.
• Consistency. These tools ensure that the documentation is consistent in style and terminology, which is crucial for maintaining the quality and clarity of user guides across different parts of the software.
• Scalability. AI-powered tools can easily update and expand documentation to keep pace with software updates and new features without starting from scratch each time.

Cons:

Among the cons, we can consider the following:

• Lack of Deep Understanding. AI may not fully grasp the nuances of user needs and the contextual use of the software, which can lead to less user-friendly documentation.
• Over-Reliance on Inputs. The quality of AI-generated documentation heavily depends on the quality of the inputs (code annotations, comments, etc.). Poor inputs can result in poor outputs.
• Generic Content. AI might generate documentation that is technically accurate but generic and not addressed to a specific audience. This means that it may be too specific or too poor, depending on the intended audience.

ConclusionsIn this article, we’ve described three AI documentation generators along with their features and a comparison among them.

While we understand the power of AI, a piece of advice we’d like to give is the following: if you’re considering using an AI documentation generator, consider also collaborating with an experienced technical Writer. In fact, as AI can speed up the process, the intervention of an experienced human is the touch that makes the portrait perfect to suites your specific needs.

The post Unlocking Efficiency: 3 AI Tools for Automatic Software Documentation appeared first on Semaphore.

View Details

On a modern site, each web page typically retrieves data by contacting online services and backends via API. Considering the popularity of the microservices approach, your site is likely to rely on several external APIs.

Calling an API in a Next.js component is simple and takes only a few lines of code. The problem is that the application may end up containing the same logic for making an API call in multiple places. This introduces code duplication issues and makes your codebase less maintainable. With the API layer approach presented in this article, you can address all those downsides.

Suppose we have a layer that contains everything needed to call all the APIs on which the site depends. Encapsulating all the logic for making API calls in the same place would make it easier to update an endpoint string or request behavior. This is the main goal of an API layer.

Let’s find out what a Next.js API layer is, why your site needs one, and how to implement it using the fetch() HTTP client.

What Is an API Layer?An API layer is the part of your Next.js Application that contains the Programming logic for retrieving data via some Interface (API) on the server. In detail, it centralizes all API calls your server components make to fetch data from external services and backends.

All the logic required to build this architectural layer is encapsulated in the api folder. This consists of several files with the following naming convention:

<externalService>API.js The <externalService> qualifier facilitates the organization of all APIs that reference the same external service or backend in the same file. Note that a Next.js API layer may also involve some utility and configuration files. These do not follow the naming convention above.

Here is what a sample API layer in a Next.js application may look like:

api├── configs│ └── ...│├── AuthorAPI.js...├── ProductAPI.js...└── UserAPI.js Now, you can make an API call in a Next.js server component using the API layer as in the example below:

import { ProductAPI } from "@/api/PokemonAPI"export default async ProductListComponent() { // retrieve the desired data using the API layer const products = await ProductAPI.getAll(); // presenting the product list ... return <div>...</div>} ProductListComponent is a server component. As such, ProductAPI.getAll() will generate a server-to-server HTTP request that will be performed by the Next.js server at page generation or build time.

Let’s now dig into the reasons why you need an API layer in a Next.js application.

Why Your Next.js Project Needs an API LayerAdopting an API layer can lead to several benefits in your Next.js architecture. Explore the three most important ones.

1. Avoid code duplicationIn a Next.js App Router application, top-level page components usually consist of several server components. These retrieve data to be rendered in the HTML by making requests to API endpoints.

As different server components may need the same data, you end up pasting the following logic pattern calling the same API throughout your codebase:

// the logic required to retrieve all products// with fetch() via APIconst productsResponse = await fetch("https://your-cms.com/api/v1/products");const productsData = await productsReponse.json();const products = productsData.results; Each time you have to retrieve the list of all products from the CMS via API, you will have to repeat those three lines of code. As you can imagine, this approach leads to code duplication.

By encapsulating API calls into reusable functions within API layer files, you can avoid writing boilerplate duplicate code. The three lines of repeated code above will become this more readable and maintainable single line of code:

const products = await CmsAPI.getProducts(); 2. Centralize all API request logic in the same placeAdopting an API layer enables the centralization of all the logic needed to call APIs in the same place, which has several advantages. For example, suppose the endpoint of an API changes (e.g., from /v1/posts to /v1/articles).

To address that change in the code with an API layer, you would just need to update the endpoint string in the fetch() call inside the specific API function. All API calls referred to that endpoint will now work as desired. This is the main benefit of logic centralization. Without an API layer, you would have to search for all occurrences of the endpoint in your codebase and manually update each one instead. What a tedious and error-prone task!

Also, the centralization offered by the API layer simplifies configuration. This means that if you suddenly need to pass new header values or cookies in some requests, you would only have to modify some API functions accordingly.

1. Deal with data caching easilyStarting with version 13, Next.js comes with a special implementation of the fetch() function from the Fetch API. In addition to all the features fetch() offers in the browser and Node.js, this implementation allows each server-side request to specify its own persistent caching behavior.

In other words, any HTTP request made with the Next.js fetch() function inside server components will produce a result that can be cached on a persistent cache on the server. This cache stores data so that it does not have to be retrieved again on the same request. The server-side caching mechanism leads to reduced network usage, improved page generation time, and improved server response time. Learn more in our guide on how to build a caching layer in Node.js.

By default, Next.js automatically caches the data returned by fetch() requests in the Data Cache on the server. This behavior depends on the optional cache option:

fetch(`https://...`, { cache: "force-cache" | "no-store" }) The possible values are:

• force-cache: Before making the request, Next.js looks for a matching response in its cache. If there is a match and it is fresh, it returns that data. Otherwise, the server performs the request and updates the cache with the updated response. This is the default value.
• no-store: Next.js ignores the cache, fetching the resource on every request and not updating the cache with the retrieved content.

Another interesting option is next.revalidate:

fetch(`https://...`, { next: { revalidate: false | 0 | number } }) This sets the cache lifetime of a resource. The allowed values are:

• false: Cache the resource indefinitely. This is the default value.
• 0: Prevent the resource from being cached.
• number: Specify the number of seconds the resource should be in the cache.

Find out more about data caching with Next.js in the official documentation.

By implementing the API layer with the Promise-based fetch() HTTP client, you can gain access to all these powerful caching options.

Implement an API Layer in Next.js With fetchFollow this tutorial section and learn how to build an API layer in a Next.js application based on the App Router. If your Next.js application uses the Pages Router instead, you can follow the same approach we introduced in guide on how to implement a Node.js API layer.

Do not forget that the API layer implemented here will be used to retrieve data when generating pages on the server. To dynamically retrieve data on the frontend, you can integrate your Next.js application with a React API layer.

Time to build an API layer in Next.js using fetch()!

PrerequisitesFirst, you need a Next.js 13+ application using the App Router approach. You can create one with create-next-app as follows:

npx create-next-app@latest During the initialization process, make sure “Yes” to the following question:

Would you like to use App Router? (recommended) » No / Yes As mentioned before, the Promise-based HTTP client used to implement the Next.js API layer will be the fetch() utility. That function is part of the Next.js core API, so you do not need to install any external dependencies.

Add the API layer to your architectureLet’s assume that your Next.js App Router project is based on the recommended /src directory. The file structure of your project will be something like this:

└── src│ └── app│ ├── favicon.ico│ ├── globals.css│ ├── layout.js│ ├── page.js│ └── page.module.css├── public│ ├── next.svg│ └── vercel.svg├── .eslintrc.json├── .gitignore├── jsconfig.json├── next.config.mjs├── package-lock.json├── package.json└── README.md Prepare your application to host an API layer by adding an api folder inside /src. The new project structure will be:

└── src│ └── api│ └── ...│ └── app│ └── ...... The api subdirectory will contain all the logic required to perform server-side API calls in your Next.js application.

Implement the API layer with caching logicSuppose your Next.js site relies on a headless CMS technology, which exposes public endpoints you need to contact to retrieve the data required to generate web pages on the server or at build time.

You can organize all those endpoints in a cmsAPI.js API layer file inside the api folder as follows:

// src/api/cmsAPI.js// common variable to use in all API object functionsconst baseUrl = "https://your-cms.com/api/v1";// other common logic...export const CmsAPI = { get: async function (slug, options = undefined) { const response = await fetch(`${baseUrl}/article/${slug}`, { method: "GET", ...options, }); return await response.json(); }, getArticles: async function ( { limit = 10, offset = 0 }, options = undefined ) { const queryParams = new URLSearchParams({ limit: limit, offset: offset, }); const response = await cmsClient.request( `${baseUrl}/articles?${queryParams}`, { method: "GET", ...options, } ); return await response.json(); }, getAllDrafts: async function () { const response = await cmsClient.request(`${baseUrl}/articles/drafts`, { method: "GET", ...options, }); return await response.json(); }, // other CMS APIs ...}; This is a real-world example of an <externalService>API.js API layer file. As you can see, it is nothing more than a JavaScript object that exposes functions corresponding to the public endpoints exposed by the CMS. In particular, each object function uses fetch() to contact a specific CMS endpoint and retrieve the desired data.

Notice that each API function accepts an optional options argument. That is required because the Next.js fetch() method supports all the options of the fetch() implementation in the Fetch API. The options argument allows the caller to customize the behavior of the API call as needed, for example by setting custom headers.

As introduced earlier, the Next.js fetch() function also accepts special options for handling the caching behavior. The optionsparameter can also be used to change that as in the following example:

// retrieve the articles from the CMS // by ignoring the data in the cacheconst articles = await CmsAPI.getArticles( { limit: 20, offset: 0, }, { cache: "no-store", }); Since <externalService>API.js files may contain server-only logic—such as basic authentication information—you do not want developers to accidentally use them in React client components.

As a rule of thumb, server components should retrieve data using the Next.js API layer, while client components should communicate with external APIs with an HTTP client like Axios or using a React API layer. Although the Next.js and React API layers serve a similar purpose, they consist of files that have different scopes.

To prevent developers from using the Next.js API layer in client-side React components, install the server-only package. Add it to the project dependencies with the following command:

npm install server-only Then, mark your <externalService>API.js files as server-only with this line at the top of the files:

import "server-only";// API layer file definition ... Now, if a developer imports a <externalService>API.js into a client component, the build process of the Next.js application will fail with the error below:

You're importing a component that needs server-only. That only works in a Server Component which is not supported in the pages/ directory. Read more: https://nextjs.org/docs/getting-started/ Note that here you saw how to implement a single <externalService>API.js file, but your Next.js API layer can contain many of them.

API layer in actionTime to see the API layer in action in a Next.js demo application that relies on the free and open-to-use PokeAPI project as an external API service provider.

Clone the GitHub repository of the demo project, install the dependencies, and launch the demo site locally with the following commands:

git clone https://github.com/Tonel/api-layer-nextjscd api-layer-nextjsnpm installnpm run dev Visit the https://localhost:3000 page in the browser and you will see:

The data presented on this page comes from the GET https://pokeapi.co/api/v2/pokemon?limit=21&offset=0 endpoint.

You can verify that the Next.js fetch() function caches data by default as expected by inspecting the cache folder inside the

.next build directory:

Open the file inside the fetch-cache subfolder and you will see that it contains a file with hashed data related to the https://pokeapi.co/api/v2/pokemon?limit=21&offset=0 endpoint.

Click on a Pokemon card to reach the detail card page:

This will trigger a new server-side API call and lead to a new cache file in the .next/cache/fetch-cache folder.

Et voilà! You just learned how to build an API layer in Next.js.

ConclusionIn this article, you understood what a Next.js API layer is, the main benefits it brings to your architecture, and how to implement it using fetch(). An API layer is a set of files that provides everything your server components need to retrieve data via API calls. It centralizes the server-side API logic in your project and makes your codebase easier to maintain. Adding this layer to your Next.js App Router application is easy and does not require external dependencies. You have now learned how to implement a server-side API layer in Next.js and saw it in action through a simple demo.

The post Why Your Next.js Site Needs an API Layer and How To Build It appeared first on Semaphore.

View Details

React 18 has been around for more than two years, and it is finally time to welcome React 19. The main innovation introduced, the one we all love is the React Compiler! It promises to forever simplify frontend development by eliminating the need for manual memoization optimizations.

In this guide, you will understand what React Compiler is, how it works, what benefits it introduces, and how to prepare your application.

What React 19 Brings To the TableReact 19, to be released at React Conf 2024 on May 15-16, 2024, is the long-awaited next version of this framework. React 18 was released in 2022, and since then, web technologies have evolved greatly. It is high time for an update.

Not only will version 19 be a step forward but it promises to forever change the way developers build applications in React. Some of the most exciting features React 19 plans to introduce are:

• Server components: Server-side rendering of components for faster page loading and better SEO. By processing components on the server before delivering the page to users, React 19 enables faster website loading times, improved search engine visibility, and smoother data management. Next.js already uses this feature.
• Actions: Streamline the management of data and interactions within web pages. Actions make it easier to update page information through forms, removing complexities and simplifying the user experience.
• Optimized asset loading: Background loading of site assets for smoother page changes. React 19 can begin loading images and other files in the background while users are still browsing the current page, reducing wait times during page transitions.
• Document metadata: Streamlined SEO management thanks to the new <DocumentHead> component. Adding titles and meta tags to pages will be more intuitive, improving search engine optimization without the need for repetitive coding.
• Web components: Improved compatibility with the Web Components standard for more flexible and compatible frontend development.
• Enhanced hooks: Greater control over the lifecycle and state of components through both existing and new hooks. The end goal is to simplify the coding process, making React development more efficient and enjoyable.
• React compiler: Automatic conversion of React code into standardized, optimized JavaScript code. It allows compiled React code to automatically render only the right parts of the UI when the state changes, reducing the need for useMemo, useCallback, and memo. This means faster React applications with simplified codebases.

All of these features are great, sure, but what stands out is React Compiler. This upgrade promises to change React development forever. Let’s now try to understand the why behind it and what benefits it will introduce!

React Compiler: Everything You Need to KnowDig into the React 19 Compiler tool, exploring what it is and how it works.

React’s Core Mental ModelTo understand the why behind React Compiler, you first need to delve into some key React concepts.

At the heart of React is a declarative and component-based mental model. In frontend development, declarative programminginvolves describing the desired end state of the UI without specifying each step to get there through DOM manipulation. Meanwhile, the component-based approach breaks down UI elements into reusable, concise, self-contained building blocks, promoting modularization and ease of maintenance.

To efficiently identify specific DOM elements that require updates, React employs an in-memory representation of the user interface called the virtual DOM. In case of changes to the application state, React compares the virtual DOM with the real DOM, identifies the minimum set of changes needed, and accurately updates the real DOM.

In short, the mental model is that React re-renders whenever the state of the application changes. However, sometimes React can be too reactive, resulting in unnecessary re-renders that significantly slow down your application.

Re-Rendering Hell: The Need for a CompilerReact’s agility in responding to changes in application state is a double-edged sword. On the one hand, it simplifies frontend development due to its declarative approach. On the other, it can lead to excessive re-rendering of the components in the UI in response of state changes.

Re-rendering problems are particularly common when dealing with JavaScript data structures such as objects and arrays. The problem is that there is no computationally efficient way to compare two objects and arrays to see if they are equivalent (have the same keys and values) in JavaScript.

Consider the following scenario. You have a React component that generates a new object or array upon each render, as in the example below:

import React from "react";const AlphabetList = () => { // define the alphabet array const alphabet = Array.from({ length: 26 }, (_, i) => String.fromCharCode(65 + i)); return ( <div> <h2>Alphabet List</h2> <ul> {/* render the alphabet as list items */} {alphabet.map((letter, index) => ( <li key={index}>{letter}</li> ))} </ul> </div> );};export default AlphabetList; Although the content of the local array will be the same at each rendering, React cannot efficiently know that. As a result, it may trigger a re-render in the nested DOM elements of the component relying on the values in that array, unaware that the UI should remain identical. This re-rendering mechanism can quickly spiral out of control, significantly impacting application performance and user experience.

To optimize re-rendering behavior and avoid those issues, React developers must manually introduce memoization into their components. In React, memoization involves caching the results of expensive computations or component outputs based on their input parameters. By storing and reusing these results, memoization helps prevent unnecessary re-reading of components, improving the overall efficiency and performance of a React application.

React 18 provides several memoization tools:

• React.memo(): A higher-order function to avoid re-rendering a component when its props remain unchanged.
• useMemo(): A React hook that caches the result of a calculation between re-renders, reducing redundant computations.
• useCallback(): A hook from React that caches the definition of a function between re-renders, avoiding unnecessary recreations of functions. Learn more in our guide to the React useCallback() hook.

Thanks to the useMemo() hook, you can optimize the <AlphabetList> component to avoid unnecessary re-renders as follows:

import React, { useMemo } from "react";const AlphabetList = () => { // define the alphabet array via useMemo() const alphabet = useMemo(() => { return Array.from({ length: 26 }, (_, i) => String.fromCharCode(65 + i)); // no dependencies, so it will only be calculated once on the first render }, []); return ( <div> <h2>Alphabet List</h2> <ul> {/* render the alphabet as list items */} {alphabet.map((letter, index) => ( <li key={index}>{letter}</li> ))} </ul> </div> );};export default AlphabetList; The memoization tools offered by React are certainly powerful. At the same time, their introduction is nothing more than a clear departure from the declarative philosophy underlying React’s core mental model. This is because the burden falls on developers, who must not only describe the final state of the UI but also explicitly manage rendering optimizations. Manual memoization also introduces code complexity and maintenance headaches.

The solution? An advanced compiler capable of converting React code into optimized JavaScript code, so that components are automatically re-rendered only in case of significant changes in state values.

What Is React Compiler?React Compiler, also known as React Forget, is an optimizing compiler for React. It now powers Instagram’s web portal in production and will be deployed on other Meta applications before its first open-source release.

The original goal of the compiler was to enforce React’s core programming model by automatically generating the equivalent of memo, useMemo, and useCallback calls to minimize the cost of re-rendering. The project has evolved greatly from its beginnings, moving from an “auto-memoizing compiler” to an “automatic reactivity compiler.”

The real objective of React Forget is now to ensure that React applications have the right amount of reactivity by default. In other words, apps should re-render only when state values meaningfully change. Currently, React re-renders a component when object identity changes. With React Forget, it will re-render only when the semantic value of an object changes — but without incurring the runtime cost of deep comparisons.

From an implementation point of view, React Compiler applies automatic memoization. However, the team behind it considers reactivity framing as a more complete way to understand what it does. If you want to learn more about the inner workings and logic of the React compiler, read the React Labs section dedicated to it.

See React Forget in action in the video below: https://www.youtube.com/watch?v=qOQClO3g8-Y&

While JavaScript is a notoriously difficult language to optimize because of its dynamic nature and loose rules, React Compiler can compile code safely by modeling both JavaScript rules and the “rules of React.” These rules limit what developers can do, helping to carve out a safe space for the compiler to perform its optimizations.

The Rules of ReactReact comes with a set of rules that are intended to support high-quality web applications. Developers should follow these rules, which also represent what React Compiler is based on.

Some of the most important rules of React are:

• Components must be idempotent: React components should always produce the same output given the same set of inputs, which include props, state, and context.
• Side effects must run outside of render: Side effects, such as data fetching, subscriptions, or manually changing the DOM, should not be performed during the rendering process of a component. Instead, they should be executed in lifecycle hooks like useEffect.
• Props and state are immutable: Props and state in React components should be treated as immutable, meaning they should not be directly modified. Changing props or state directly can lead to bugs and unpredictable behavior.
• Return values and arguments to hooks are immutable: Once values are passed to a React hook, they should not be modified. Hooks rely on the immutability of their arguments and return values to maintain consistency and predictability in component behavior.
• Values are immutable after being passed to JSX: Do not mutate values used in JSX after this has been rendered. Any necessary mutations should be performed before the JSX is created to ensure that the rendered output remains consistent.
• Never call component functions directly: React components should only be used within JSX and not called directly as regular functions.
• Never pass around hooks as regular values: React hooks, such as useState and useEffect, should only be called inside functional components. Using them as regular values can lead to unexpected behavior and violate the rules of hooks.
• Only call hooks at the top level: React hooks should always be called at the top level of functional components, before any conditional statements or loops. This ensures that hooks are called in the same order on every render and maintain their intended behavior.
• Only call hooks from React functions: Hooks should only be called from within React function components or custom hooks. Calling hooks from regular JavaScript functions can lead to errors and violate the rules of hooks.

Enable Strict Mode and configure React’s ESLint plugin to make sure that your React application follows these rules.

Benefits and HopesThe main benefits introduced by React Compiler are:

• No more memoization headaches: Developers no longer have to manually implement and manage memoization strategies in their code. This reduces complexity and the likelihood of errors, streamlining the development process.
• Better developer experience: Developers can focus more on building features and less on performance optimization, leading to increased productivity and satisfaction. They will finally be able to fully embrace the React declarative approach.
• Faster React applications: Render components only when necessary, minimizing unnecessary computations and overhead. This results in faster and more responsive user interfaces, improving overall performance and user experience.

These are all promising changes, but we have yet to see what effect this new tool will have on code development. To ensure that the compiler does its job, your code must follow the rules of React. Here is why the official team recommends adopting ESLint and similar tools to prepare your application for React Compiler.

React Compiler: A Frontend Revolution?It is difficult to say soon whether the React Compiler will be enough to spark a real revolution in the realm of frontend development. What is for sure is that it has all the credentials to significantly change the development of future React applications. By introducing automatic memoization, this compiler can automatically speed up React applications and improve the developer experience. These are just some of the benefits this promising ambitious project brings to the tabl.

React Conf 2024 is just around the corner, and we look forward to seeing what breakthroughs this ambitious project will have on the frontend world!

The post React Compiler: What Is It and How Will It Change Frontend Development? appeared first on Semaphore.

View Details

A crucial step in the software creation process is effective testing. It ensures the programs’ general quality, security, and usefulness. Testing teams do, however, have a number of difficulties, one of which is finding faulty tests. Sometimes these tests fail for no obvious reason at all, which confuses people and impedes the progress of development. These tests are called flaky tests. This article covers various testing tools that can help you spot, control, and minimize the effect of flaky tests.

Challenges in Identifying Flaky TestsIn software testing, random test failures may be very frustrating. Flaky tests are unexpected since they can pass on one try but fail on another. These inconsistencies might be caused by difficult-to-replicate external dependencies, complex links between many code parts, or timing problems. Finding incorrect tests becomes more difficult when several tests are run simultaneously, application behavior changes, and unexpected test execution conditions occur.

The asynchronous and dynamic loading of web application components makes the job more difficult. As a result, it is difficult to create a reliable and consistent testing environment and find the root cause of random test failures. This is when certain testing tools become important. By resolving these problems, you may improve the dependability of your test results using these tools. They allow testing teams to identify, control, and mitigate the effect of irregular tests on the overall testing strategy. You can read more about the best practices to help mitigate it here

Tools for Detecting and Managing Flaky TestsSeveral testing tools can simplify the process of finding and handling flaky tests. Let’s explore some famous options:

TestNGTestNG, a famous Java testing framework, includes built-in features for handling flaky tests. It uses listeners and analyzers to closely watch test runs, discover problems, and properly group tests.

• Listeners: These are important for recording events during a test run. They offer hooks for running custom code before or after defined test events. Listeners allow you to create custom rules for identifying and handling unreliable tests.
• Analyzers: TestNG analyzers have a built-in way for rerunning failed tests, which helps decrease flakiness. The analyzer checks the original test results, finds failures, and selectively retries the failed tests based on predefined criteria.

Adding TestNG to your project

Here is a step-by-step guide on installing TestNG in IntelliJ IDEA (it comes bundled with TestNG 7.1.0)

• For Maven users:
• Navigate to your project’s root directory.
• In pom.xml, hit Alt Insert and choose Dependency.
• In the window that appears, enter testNG into the search area.
• Locate the org.testng:testing dependency, choose its version from the search results, and then click Add.
• After adding the dependency to pom.xml, import the modifications by pressing Ctrl Shift + O or clicking the refresh icon to import the changes.
• For Gradle users:
• Open the build.gradle in your project’s root location.
• Press Alt Insert in build.gradle and choose Add Maven artifact dependent.
• Enter testNG into the search area of the window that opens.
• Find the requirement for org.testng:testing, choose its version from the search results, and click Add.
• Once the dependent has been added to build.gradle, import the changes by pressing Ctrl Shift O or refresh icon to import changes
• For IntelliJ build tool:
• Select File then Project Structure from the main menu or use Ctrl Alt Shift S
• Under Project Settings, choose Libraries, and then click the New Project Library from Maven option.
• In the dialog that appears, specify the required library asset, for example: org.testng:testng:6.14.3.
• Apply your changes and close the dialog.

Creating a new TestNG class

The simplest way to make a new test class in IntelliJ IDEA is to use a special intention action that can be called from the source code. In this case, the IDE makes a new test class and produces test code for it, the package, or function. Here are steps to take:

• Place the caret at the class for which you wish to build a test in your production code in the editor, hit Alt Enter, and choose build Test.

• Choose the library you wish to use from the Create Test dialog box. It will ask you to download the required library if you don’t already have it. Click Fix to accomplish it.

• The IDE will add the missing dependencies to your pom.xml if you’re using Maven. Manually add the required dependencies for projects using Gradle.

• Set the name and location of the test class and choose the methods you wish to test. Press OK.
• Consequently, a new test class with the given name and produced test methods are created under the Test Sources Root by IntelliJ IDEA.

Running TestNG tests Executing TestNG tests within IntelliJ IDEA is straightforward:

• Individual Test: Click the “Run” icon in the gutter next to the desired test.
• Entire Test Class: Right-click the test class name and select “Run” -> “Run.” The “Run” tool window will display the test results for your evaluation.

JUnit Flaky Test PluginThe JUnit Flaky Test Plugin is an extremely useful tool for developers working with JUnit test packages. It handles a prevalent problem which is flaky tests. These are tests that give inconsistent results, with some passing and others failing for no obvious reason. This difference can hinder development progress by wasting time studying mistakes that do not suggest true issues. JUnit 5 is the latest version of the JUnit testing system, offering a contemporary base for developer-side testing on the Java Virtual Machine (JVM). This includes emphasis on Java 8 and higher, as well as allowing for a range of testing methods. Here is how to plan and perform tests.

Creating Project

1. Navigate to file, select new and project from the main menu.
2. Choose Java from the left selection in the New Project wizard.
3. Give the project a name, such as `junit-tutorial , and choose Maven, Gradle or IntelliJ as the build tool.
4. Choose the JDK that you wish to utilize for your project from the list of JDKs.
5. Choose Add JDK and enter the path to the JDK home directory if the JDK is installed on your machine but is not defined in the IDE.
6. Click Download JDK if your machine is missing the required JDK.
7. Press the Create button.

Add dependency

We must add JUnit as a dependency in order for our project to leverage JUnit functionalities.

• For Maven users:
  • In your project’s root directory, open pom.xml.
  • Press Alt Insert in pom.xml and choose Dependency.
  • Enter org.junit.jupiter:junit-jupiter in the search box of the dialog that appears.
  • In the search results, choose the required dependency and select Add.
  • Once the dependency has been added to pom.xml, import the modifications by pressing Ctrl Shift O or selecting refresh to import changes.
• For Gradle users:
  • Launch the build.gradle in your project’s root directory. Ctrl Shift 0 may be used to swiftly go to a file by entering its name.
  • Press Alt Insert in build.gradle and choose Add Maven artifact dependency.
  • Enter org.junit.jupiter:junit-jupiter in the search field of the tool window that appears.
  • In the search results, choose the required dependency and select Add.
  • Once the dependent has been added to build.gradle, import the modifications by pressing Ctrl Shift O or clicking refresh to import changes
• For IntelliJ build tool:
  • Select File, Project Structure (Ctrl Alt Shift S) from the main menu.
  • Choose Libraries from the Project Settings menu, then click the New Project Library from Maven button.
  • In the resulting dialog box, choose the required library artifact, such as: org.junit.jupiter:junit-jupiter:5.9.1.
  • After making your adjustments, close the dialog box.

Creating Sample Code and Tests

In the Project tool window go to src/main/java and create a Java file called Calculator.java.

import java.util.stream.DoubleStream;public class Calculator { static double add(double... operands) { return DoubleStream.of(operands) .sum(); } static double multiply(double... operands) { return DoubleStream.of(operands) .reduce(1, (a, b) -> a * b); }} Right-click on “Calculator” in the Project tool window and select “Create Test.” Choose the two methods you want to test (add and multiply).

The editor takes you to the newly created test class. Modify the add() test as follows:

@Test@DisplayName("Add two numbers")void add() { assertEquals(4, Calculator.add(2, 2));} This short test will determine whether our algorithm adds two and two correctly. The test has a more practical and useful name thanks to the @DisplayName annotation. What would happen if you wanted to include more than one statement in a single test and run it all even if some of the assertions didn’t pass? The assertAll() function accepts a list of lambda expression assertions and checks that each one has been verified. Seeing a specific result instead of the test’s overall result is always more handy than having several single assertions.

Run tests

After we have set up the code for the testing, we can run the tests and find out if the tested methods are working correctly. Running individual test, click run in the gutter. To execute all tests in a test class, click execute against the test class declaration and then choose Run. The final result can be seen below:

RSpec’s –bisect OptionA potent method called –bisect is given by the well-known Ruby testing framework RSpec to overcome inconsistent test failures. When tests only fail after finishing particular tests before them, a situation that might be difficult to fix using standard methods, this flag becomes extremely important.

A repeating method is invoked by the --bisect option to find the source of test flakiness. It starts by running your whole test suite, dividing it in half carefully and running each half separately. Subdivision continues until the smallest set of tests successfully reproduces the mistake in the half that still shows the failure.

Practical Example Consider a Ruby test suite with ten files (spec/calculator_1_spec.rb to spec/calculator_10_spec.rb), and one of them occasionally fails. You assume the failure is caused by the execution order, but you can’t determine the specific combination.

Using --bisect:

1. Open your terminal and navigate to your Ruby project directory.
2. Run the following command to start the bisect process:

rspec --seed 1234 --bisect --seed 1234: This guarantees that tests are executed in a consistent sequence during the bisect process. RSpec will provide findings that include rounds of bisection, eventually narrowing down the failed samples. Here is a sample output:

Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3Round 4: bisecting over non-failing examples 1-2 .. ignoring example 1Bisect complete! Reduced necessary non-failing examples from 9 to 1. Success! RSpec has identified the minimal set causing the failure. The final output provides a minimal reproduction command that includes only the necessary failing specs. This command below allows you to re-run the failing tests in isolation, making it easier to diagnose the root cause.

rspec ./spec/calculator_10_spec.rb [1:1] ./spec/calculator_1_spec.rb [1:1] --seed 1234 You can use Ctrl-C to end the bisect operation at any time. You can test it with projects here

Visual Regression Testing ToolsVisual Regression Testing (VRT) tools are crucial in detecting even the slightest visual differences that could be overlooked during conventional testing. In this chapter, we will review Percy and Applitools, two prominent players in the VRT industry. Both platforms provide unique benefits that are worth exploring.

Overview of Percy’s Visual Testing Capabilities

The purpose of Percy, an automated visual testing tool, is to identify visual inconsistencies in online applications. It helps teams ensure the visual excellence of their user interface by capturing and comparing screenshots of the application’s visual state during testing. Jest, Cypress, and Selenium are just a few of the well-known testing frameworks that Percy easily interfaces with. Additionally, it works with CI/CD systems like GitHub Actions and Jenkins. It creates baseline images of the user interface for your programme and notifies you of any differences by comparing subsequent test runs with these baselines.

Integrating Percy with Jest:

• Begin by installing the Percy package in your project.

npm install --save-dev @percy/puppeteer * Set up Percy in your Jest configuration.

// jest.config.jsmodule.exports = { // ... other Jest configurations setupFilesAfterEnv: ['<rootDir>/percy.setup.js'],}; * Create a percy.setup.js file with the following content:

// percy.setup.jsconst { percySnapshot } = require('@percy/puppeteer');// Ensure Percy is runningbeforeAll(async () => { await page.goto('http://localhost:5338');});// Take snapshots using Percytest('Snapshot', async () => { await percySnapshot(page, 'Snapshot');}); * Run Jest with Percy: Execute your Jest tests with Percy.

npx percy exec -- jest The integration with Percy enhances your visual testing strategy, providing a comprehensive view of your application’s UI changes.

Integration with Selenium:

If you’re working with Java and Selenium, you can use the percy-java-selenium library in your project. Add it to your Maven project dependencies

<dependency> <groupId>io.percy</groupId> <artifactId>percy-java-selenium</artifactId> <version>1.3.0</version></dependency> A sample of test code in Java for taking snapshot:

import org.openqa.selenium.WebDriver;import org.openqa.selenium.chrome.ChromeDriver;import io.percy.selenium.Percy;public class Example { public static void main(String[] args) { // Create a new WebDriver instance using ChromeDriver WebDriver driver = new ChromeDriver(); // Navigate to the specified URL (in this case, "https://example.com") driver.get("https://example.com"); // Initialize Percy with the WebDriver instance Percy percy = new Percy(driver); // Take a snapshot of the current page with a descriptive name percy.snapshot("Java example"); }} In short, this code sets up Percy, starts “https://example.com“, sets up a Chrome WebDriver, and takes a picture for testing and comparison. Percy will keep this sample for future use as a guide and for checking for bugs. Keep in mind that you should use the real URL of the site you want to test instead of “https://example.com.” You can test it with various Java projects here

Applitools Eyes: Visual AI Testing

One of the most complete Visual AI testing tools is Applitools Eyes, which has a lot of options for finding UI issues. Users pay based on how many tests they run and the services they need, using a commercial price plan. Test scripts for Applitools Eyes can be written in Python, Java, and JavaScript, among other languages. The Applitools SDK is used to compare snapshot that were taken while tests were running. In particular, Applitools Eyes cuts down on false results by using AI and machine learning. It ensures accurate test results by instantly finding and disregarding changing visual aspects.

To start using Applitools Eyes for Visual AI Testing, follow these steps:

• Sign Up for an Account: Visit the Applitools sign-up page to create a new account.
• Login to Applitools Dashboard: Once registered, log in to the Applitools dashboard.
• Set Up the SDK in Your Project: To start using Applitools, you’ll need to install and configure the Applitools SDK. The SDK is available for popular programming languages and frameworks and can be easily integrated to existing automation test frameworks. Each Applitools account has a secret API key. Visual tets require the API key to authenticate the account and upload test results to the Applitools cloud and connect the results to that account
• Write Your First Visual Test: Using Applitools Eyes, create a test script that defines visual checkpoints. This entails taking screenshots of your application and confirming their accuracy.

Example (Java with Selenium):

import com.applitools.eyes.selenium.Eyes;import org.openqa.selenium.WebDriver;import org.openqa.selenium.chrome.ChromeDriver;public class YourVisualTest { public static void main(String[] args) { // Initialize the WebDriver for Chrome browser (you can modify this to use a different browser) WebDriver driver = new ChromeDriver(); // Create an Eyes object to perform visual testing Eyes eyes = new Eyes(); // Set your Applitools API key to enable communication with the platform (replace with your actual key) eyes.setApiKey("Your_Applitools_API_Key"); try { // Start the visual test by providing the application name and test name of your choice eyes.open(driver, "Your App Name", "Your Test Name"); // Navigate to the URL of your web application under test driver.get("Your App URL"); // Capture a visual checkpoint of the entire page. // Consider using more specific selectors for targeted testing in the future eyes.checkWindow("Full Page"); } finally { // Always close the WebDriver to avoid resource leaks driver.quit(); // If the test was aborted due to an exception, abort the Eyes test as well eyes.abortIfNotClosed(); } }} * Run Your Visual Test: When you run your test script, Applitools will take screenshots of your application and compare them to a baseline. * Review Results in the Dashboard: Return to the Applitools dashboard to see the visual tests’ outcomes. It will draw attention to any variations between your application’s baseline and current states. Update the baseline in the Applitools dashboard to reflect the new intended visual design as your application develops.

Adding visual testing to your workflow guarantees a visually consistent and enjoyable user experience, regardless of whether you want the powerful AI-driven analysis offered by Applitools or the seamless integration, collaboration, and ease of use offered by Percy.

ConclusionThis article discusses the challenges faced by flaky tests in software development, with a particular emphasis on the role of testing tools in resolving these issues. To study more of this tools and practical experiences in large-scale commercial settings and investigating technologies such as TestNG, JUnit, and Percy, check out this in-depth study. It becomes clear that the testing tools used have a substantial impact on test suite dependability. The study emphasizes the ongoing need for research and innovation in testing technologies to improve their ability to detect and manage flaky tests. As the software development community grows, a concentrated effort to refine and advance testing tools is critical for establishing confidence, assuring reliability, and successfully negotiating the complexities of flaky testing scenarios.

The post Testing Tools to Identify Flaky Tests appeared first on Semaphore.

View Details

Forget fancy titles and complex frameworks. The secret to an organization’s success might lie in a surprisingly simple act: two people moving a couch. This seemingly mundane task, according to Gene Kim, holds profound lessons about communication, problem-solving, and creating a frictionless work environment.

Edited transcriptionFor over 25 years, author and researcher Gene Kim has studied the intricate dynamics of high-performing technology organizations, including technical practices, architectural approaches, and cultural norms that allow companies to operate at their peak. Gene’s latest book, Wiring the Winning Organization co-authored with Dr. Steven Spear, collects these rich learnings into a framework for optimizing how teams interact and enable their workflow.

The Couch Metaphor Explained: Illuminating organizational principles, team coherence, and DevOpsAt the core of Wiring the Winning Organization lies a deceptively simple yet profound metaphor “that I think is probably one that has changed the way I view the world,” Gene says. The couch metaphor involves two individuals, Steve and Gene, deciding how to move a couch. The act of two people moving a couch may seem, at first, devoid of deeper significance and complexity, being, apparently, just brute manual labor. However, multiple layers of essential problem-solving skills are potentially required to complete this rudimentary task. As Gene explains, ” They have to solve a lot of problems, like where’s the center of gravity, to get through a narrow doorway around which axis they rotate, to get through a narrow winding set of stairs, who goes first?”

As such, the couch metaphor chiefly illustrates teams’ necessity to navigate obstacles: rapid experimentation, trial-and-error, fast feedback loops, communication, and coordination. As Gene describes, “You don’t need consultants, you don’t need focus groups. Just by picking up the couch—fast feedback, experimentation, communication, coordination—they’re going to figure out how to solve the problem.”

Likewise, the metaphor also shows how artificial constraints can severely disrupt this coherence between actors. What’s more, such disruption is often imposed by leadership, says Gene, “as leaders do things that make their work more difficult: We can turn off all the lights—it takes longer, more dangerous, can damage the couch themselves—, we introduce a lot of background noise, or put in an intermediary between Steve and Gene so they can’t talk to each other directly.” At its core, the couch analogy underscores leaders’ responsibility to figure out the workflow. Rather, “their job is to set up the architecture so everyone has what they need, when they need it, in the right format and time,” Gene states. “The opposite is that important failure signals are suppressed or extinguished, leading to catastrophic outcomes,” he adds.

A key part of understanding this architecture, as outlined in the book, is recognizing three distinct layers: the object being worked on (like code), the tools and instruments used (IDEs, etc.), and crucially, the overarching organizational wiring that dictates how components interact. “In any good to great transformation, the only thing that changes is layer three—the management system, the wiring,” Gene explains. Even though it might be the “same people, same technology, same equipment”, the wiring is what enables or obstructs high performance.

In the software development domain, the couch metaphor can also illustrate the role of DevOps. The “ever-increasing number of functional specialties” such as dev, QA, security, and operations has “partitioned the couch so they could not effectively communicate and coordinate.” To solve this issue, DevOps, through practices like CI/CD and automated deployments, provides a “different wiring” that favors cross-functional collaboration, helping “to better couple those components together and decouple them.”

Besides, as new domains like MLOps and AI Ops are integrated, Gene sees immense value in established frameworks: “It’s about creating governance processes to make sure we don’t get the company into trouble…It’s just a great time because it’s this community being handed the most important responsibilities.”

Embracing functional specialtiesWhile some may wish for a retreat from increasing specialization, Gene is confident that “functional specialties are not going to go away.” Functional divides are inescapable for solving complex problems and despite it being impossible for a single individual to be proficient in all fields, specialization is what delivers the greatest results.

The solution, therefore, isn’t eliminating specialties but rather optimizing how they integrate. Behind this integration, says Gene, is “the functional owner,” who is “the product owner of the thing you are integrating their work into.” By separating responsibilities around methodology from high-level objectives, specialized groups can maintain autonomy while still operating coherently.

The bottom lineWant to delve deeper into Gene’s work? Here are some starting points:

• You can find more information about Gene Kim, his books, and research on his official website
• Aside from Wiring the Winning Organization, his books include:
  • The Phoenix Project: A novel about IT, DevOps, and Helping Businesses Succeed. Co-authored with Kevin Behr and George Spafford, The Phoenix Project follows an IT manager as he takes over the struggling operations team at Parts Unlimited and attempts an audacious turnaround.
  • The Unicorn Project: A follow-up to The Phoenix Project, focusing on the development side of IT
  • The DevOps Handbook: Co-authored with Patrick Debois, John Willis, and Jez Humble, providing insights into creating world-class agility, reliability, and security in technology organizations.
• You can follow Gene Kim on his X account @geneskim.

The post Gene Kim on Unpacking Organizational Wiring appeared first on Semaphore.

View Details

In the context of machine learning (ML), flaky tests can be troublesome as they introduce uncertainty into the evaluation process, potentially leading to incorrect conclusions about model performance or behavior.

In this article, we’ll:

• Introduce what flaky tests are.
• Discuss issues caused by flaky tests in ML.
• Consider and Evaluate flakiness in ML.
• Discuss strategies to avoid flaky tests in ML.

Introduction to Flaky TestsFlaky tests refer to automated tests that produce non-deterministic results, meaning they may pass or fail inconsistently under the same conditions, without any changes to the code. In other words, they’re unpredictable and can pass or fail when re-run, even if there have been no modifications to the code base, dependencies, or test environment.

In the context of ML, flaky tests can manifest in various forms, causing issues that include, among others:

• Unstable model performance metrics.
• Inconsistent validation results.
• Unpredictable behavior during training or inference.

Issues Caused by Flaky Tests in Machine LearningDue to their nature, flaky tests introduce two different sets of issues in the context of machine learning:

• Infrastructure-related issues.
• Model-related issues.

Let’s discuss both of them.

Infrastructure-related IssuesIn ML environments, flaky tests can introduce several significant issues on the side of the infrastructure, complicating the development, deployment, and maintenance of ML models.

We can recall the following:

1. Resources drain. Investigating flaky tests requires significant time and computational resources. Teams often need to run tests multiple times to determine whether a failure is due to a real bug or just flakiness. This process can consume computational resources, especially with complex ML models that require substantial time and hardware to train and evaluate.
2. Development slowdown. Flaky tests can significantly slow down the development cycle. Every time a test fails inconsistently, developers and data professionals might need to interrupt their current work to investigate whether the failure indicates a real issue due to the code or model. This disruption can delay the introduction of new features or the resolution of existing bugs.
3. Impact on continuous integration/continuous deployment (CI/CD) pipelines. In a CI/CD environment, automated tests trigger actions such as building, testing, and deploying applications. Flaky tests can cause interruptions in the pipelines, requiring manual intervention to proceed. This defeats the purpose of automation, leading to inefficiencies and potential delays in deployment schedules. In addition, re-running pipelines adds to your monthly bill.
4. Reduced trust in testing frameworks. Consistent reliability in test results is crucial for the development team’s confidence in their testing framework. Flaky tests undermine this trust, making it challenging to distinguish between genuine failures that need attention and random failures that can be ignored. Over time, this can lead to a scenario where test failures are dismissed without investigation, potentially allowing real defects to slip through.

Model-related issuesFlaky tests can significantly impact various aspects of the machine learning model lifecycle, including training, validation, and generalization. These impacts can compromise the model’s performance, its reliability in production environments, and ultimately the project’s success.

We can recall the following:

1. Unreliable model evaluation. Flaky tests can lead to unreliable assessments of model performance, making it challenging to accurately determine the effectiveness of ML algorithms accurately.
2. Inconsistent validation. Flakiness in validation procedures can obscure the true generalization capabilities of ML models, potentially leading to overfitting or underfitting. This inconsistency can emerge from tests that depend too much on specific data subsets or initialization parameters that vary across runs.
3. Reduced confidence in results. Flaky tests can lead to both underestimation and overestimation of model performance on validation datasets. This makes it challenging to make informed decisions about whether a model is ready for deployment, also contrasting decision-making processes based on model outputs.
4. Difficulty in hyperparameter tuning. The process of hyperparameter tuning relies on consistent feedback to guide the search for optimal model configurations. Flaky tests introduce noise into this feedback, potentially leading to suboptimal tuning decisions based on misleading test outcomes.
5. Compromised generalization ability. The ultimate goal of an ML model is to perform well on unseen data. Flaky tests, by obscuring the true performance of a model on validation sets, can lead to overconfidence in its generalization ability. This might result in deploying models that perform poorly in real-world scenarios.

Examples of Flakiness in Machine LearningIn this section, I want to consider flakiness due to ML models themself. In other words, in machine learning, flakiness also exists due to the statistical nature of the models.

In the context of machine learning, in fact, flaky tests can arise due to various reasons such as:

• Non-deterministic algorithms.
• Random initialization.
• Data shuffling.

Let’s see an overview of some Python examples demonstrating common scenarios where flakiness might occur in ML.

Example 1: Flakiness Due to Random InitializationLet’s consider the following code:

import numpy as npfrom sklearn.datasets import make_classificationfrom sklearn.model_selection import train_test_splitfrom sklearn.ensemble import RandomForestClassifier# Generate synthetic dataX, y = make_classification(n_samples=1000, n_features=20, random_state=42)# Split data into train and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Train Random Forest classifierclf = RandomForestClassifier(n_estimators=100, random_state=np.random.randint(100))clf.fit(X_train, y_train)# Evaluate classifieraccuracy = clf.score(X_test, y_test)print("Accuracy:", accuracy) In this example, the accuracy of the Random Forest classifier varies across runs due to the random_state parameter used for initialization.

In this case, setting random_state=42 in the make_classification() function results in an accuracy of 0.895 approximately.

Instead, setting random_state=0 make_classification() results in an accuracy of 0.945 approximately.

To verify the different results that the modification of this parameter leads to, good tests involve:

• Trying different values in the make_classification() (which generates the classification data) and train_test_split()(which splits the generated data into the train and test set) functions.
• Not specifying the value of the parameter. This way, each time you run the model, you get different values because the values of the random_state parameter are set randomly.

Example 2: Flakiness Due to Data ShufflingLet’s consider the following scenario:

from sklearn.datasets import load_irisfrom sklearn.model_selection import KFoldfrom sklearn.svm import SVC# Load iris datasetiris = load_iris()X, y = iris.data, iris.target# Initialize SVM classifierclf = SVC(kernel='linear', random_state=42)# K-fold cross-validationkf = KFold(n_splits=5, shuffle=True, random_state=42)accuracies = []for train_index, test_index in kf.split(X): X_train, X_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index] clf.fit(X_train, y_train) accuracy = clf.score(X_test, y_test) accuracies.append(accuracy)mean_accuracy = np.mean(accuracies)print("Mean Accuracy:", mean_accuracy)print(accuracies) This results in:

Mean Accuracy: 0.9733333333333334[1.0, 1.0, 0.9666666666666667, 0.9333333333333333, 0.9666666666666667] In this example, the K-fold cross-validation produces different accuracy values across runs due to the shuffling of data during each split, leading to flakiness in test results.

Also, in such cases, every time we run the code, the value of accuracy we get may be different because of the shuffle=Trueparameter in the KFold() function.

Example 3: Flakiness Due to Algorithm VariabilityLet’s consider the following:

from sklearn.datasets import load_irisfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split# Load Iris datasetiris = load_iris()X, y = iris.data, iris.target# Introduce flakiness due to randomness in bootstrappingaccuracies = []for _ in range(10): # Split data into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=None) # Train Random Forest classifier clf = RandomForestClassifier(n_estimators=100) clf.fit(X_train, y_train) # Evaluate classifier accuracy = clf.score(X_test, y_test) accuracies.append(accuracy)print("Mean Accuracy:", sum(accuracies) / len(accuracies)) In this case, every time you run the code you get a slightly different value of the accuracy. This is due to variability in the algorithm’s process that, in this particular case, is due to randomness in bootstrapping that estimates the distribution of a statistic.

Note that flakiness due to the algorithms’ variability involves different ML models (not only the Random Forest).

Strategies to Avoid Flaky Tests in Machine LearningAs we’ve considered flakiness due to ML models’ management, in this paragraph we’ll consider some strategies to avoid it.

Strategy 1: Ensure ReproducibilityOne strategy to avoid flakiness in ML is to ensure reproducibility across ML experiments.

To do so, you can consider the following:

• Set the random_state parameter. The random_state parameter in scikit-learn serves as a seed for the random number generator. It ensures that you get the same results each time you run a piece of code, which is crucial for reproducibility in machine learning experiments. Common choices are integers like 0, 42, or any other fixed integer. In particular, if you’re comparing different algorithms, preprocessing steps, or model parameters, use the same random_state across all the experiments to ensure that the differences in results come from the changes you’re investigating, not the randomness in data shuffling or initialization.
• Version controlling data and code. Setting a fixed value of the random_state parameter allows you to study for flakiness due to other factors. Good tests can be made on the actual data at your disposal, as well as on the code you’ve written. To do so, a good practice is to create different experiments and freeze them, using version control for each different experiment: this way you can investigate different scenarios being sure to save everything. Finally, consider that there are also AI tools that can help you with version control like DVC.
• Documenting the experiments. To ensure you remember your choices during experiments, a good practice is to document everything you did and why you did it. This way, the reproducibility is also granted on the side of the process.

Strategy 2: Stabilize The Training ProcessThe training process can introduce flakiness if you don’t follow standardized procedures. A strategy that eliminates flakiness can consider the following:

• Hyperparameters tuning. Fixing and validating hyperparameters is a procedure that should always be followed. Leaving models with untuned hyperparameters leads to not reproducible tests and experiments.
• Using consistent preprocessing techniques. To ensure reproducibility and avoid flakiness, data preprocessing is a must to ensure the data you have does not affect randomness. Feature Scaling, for example, ensures a consistent scaling of the features across different runs by using techniques like Min-Max scaling or Standardization (if you’re not familiar with the concept of scaling features, you can read this article).
• Validate models across multiple runs. As flakiness is distributed across multiple processes and parameters in ML, validating models across multiple runs ensures reproducibility. To do so, a common choice is to use cross-validation for hyperparameter tuning. For example, k-fold cross-validation should be performed with the same number of folds and random_state across multiple runs. This helps in obtaining stable estimates of model performance by averaging results over different data splits.

Strategy 3: Implement Quality Assurance PracticesEnsuring the reliability and stability of ML workflows is fundamental to mitigating flakiness in ML. In this scenario, a good practice is to introduce quality assurance best practices to identify potential sources of variability and randomness in ML experiments.

For example, you can consider the following:

• Unit testing for data preprocessing. Develop unit tests to validate data preprocessing steps, including feature scaling, encoding, and handling missing values. By testing these components individually, in fact, you can ensure consistent data transformations across different runs, reducing the risk of flakiness due to data preprocessing variability.
• Integration testing for model training and evaluation. Conduct integration tests to validate the end-to-end ML pipeline, encompassing model training, validation, and evaluation stages. Integration tests, in fact, simulate real-world scenarios by validating the interaction between different components of the ML system, such as data loading, model fitting, and performance evaluation.
• CI/CD pipelines for automated validation. Implement CI/CD pipelines to automate the validation process across multiple runs of ML experiments that integrate unit tests and integration tests to continuously monitor the stability and consistency of ML workflows. By automating the validation process, in fact, you can detect flakiness in the early stages of the development cycle and ensure the reproducibility of results across different environments. A good way to implement such a solution is to use the Semaphore flaky dashboard
• Monitoring and logging for anomaly detection. Implement monitoring and logging mechanisms to track the behavior of ML models in production environments by continuously monitoring key performance metrics and logging anomalies or deviations from expected behavior. By proactively detecting anomalies, in fact, you can identify potential sources of flakiness and take corrective actions to maintain the stability of ML systems.

ConclusionsIn this article, we’ve shown how flaky tests can affect ML models.

Due to the stochastic nature of the ML models, there are a lot of ways to reduce and mitigate the effect of randomness. To do so, we’ve investigated how ML models can be affected by randomness and strategies to reduce it, by standardizing processes and procedures.

The post Flaky Tests in Machine Learning: Challenges and Countermeasures appeared first on Semaphore.

View Details

Ever spent hours writing a test, only to have it fail mysteriously the next day? If yes, you are not alone, It is sincerely a very common frustration for developers of all experience levels.

Let me share a little of my experience. Right before an update for our app, we ran our weekend tests to ensure everything worked fine. While earlier tests for purchasing items worked fine, on Sunday, they suddenly failed. After looking into it, we figured out that our tests were dependent on the response from a mock payment , programmed to mimic weekdays only, overlooking weekend transaction delays.

We never noticed this before because we never ran these tests on a weekend. It was a good lesson that our tests need to be more like real life, including what happens on weekends, to catch problems like this.

GoalsUnderstanding flaky tests is coming to terms that your application could be all perfect and still fail a test, it doesn’t matter if it has passed any previously, it could fail if it wants to, there are certain reasons for this, and there are certain ways we could enhance our test stability, one of which is by implementing test doubles and knowing when exactly to implement it, and thats the sole purpose of this article.

What is Flaky Test and Why Test Doubles?Similar to how stunt doubles do dangerous work in movies, we use test doubles to replace troublemakers and make tests easier to write.~Jani Hartikainen

A test that exhibits both passing and failing outcomes under the same configuration is known as a Flaky test. This tends to be very frustrating. Enhancing test stability and managing Flaky tests becomes secondary in a software development cycle.

Flaky test reduces confidence in the testing team and leads to wasted time and resources. A few of the most common causes of flakiness are Concurrency Issues, Dependency on External Services, leaked state, Platform or environmental differences, randomness, and fixed time dependency.

Employing the use of test doubles is a good choice to improve test stability, and make your tests more reliable.

They help by taking the place of unpredictable parts of your code, like external services, so your tests don’t fail unexpectedly. This approach saves time and makes sure your tests really check what they’re supposed to.

Test doubles are assumed objects that imitate the behavior of real components in your application. In other words, Test doubles are like fake parts in a system that act like the real ones, but in a way we can control.

They’re used in tests to take the place of real parts to make sure tests run the same way every time. “Test double” covers different kinds: stubs, mocks, fakes, spies, and dummies.

Types of Test DoublesTest doubles could be confusing at times and misplaced on when to be used and more especially what to be used.

Dummy ObjectsDummy Objects also known as Dummies, are usually put into parameters or arguments. These are passed around to satisfy the API but never actually used. For better understanding, they are like placeholders or fill-ins for parts of your code that need something to work, but you don’t use those parts for anything important.

Think of them as the extras in a movie scene they’re there to fill the space, so everything looks right, but they don’t have lines or impact the scene. They help your code run without errors when it’s expecting certain pieces to be there, even though those pieces aren’t doing anything.

Here is a basic implementation of a Dummy in Javascript :

const dummyObject = {};// Test using the dummytest('dummy test', () => { // Dummy is not used in this test}); In the code example above, we defined a dummy object named ‘dummyObject’, which could be usually empty. The purpose of this object is to fulfill the parameter requirement, without necessarily having any effect.

It is important to note that using a dummy object doesn’t directly address flakiness, but it helps ensure that the test remains consistent and predictable.

Spy ObjectsSpies are stubs that record some information based on how they were called. They can be used to verify that certain methods were called. In testing, you can liken spies to undercover agents, they watch and register what happens in the code without really getting involved. While they can mimic certain actions like stubs, their main job is to keep track of how and when they’re called.

Let’s look at a simple implementation of Spies:

// TEST WITH JESTconst inventoryService = { checkStock: () => Promise.resolve(true),};test("spy test", async () => { const spy = jest.spyOn(inventoryService, "checkStock"); const isInStock = await inventoryCheck("item123", inventoryService); expect(spy).toHaveBeenCalledWith("item123"); expect(isInStock).toBe(true); spy.mockRestore();}); Inside the test above, we use jest.spyOn() to create a spy on the checkStock() method of inventoryService. The spy records function calls and makes sure the behavior always resolves to true.

By ensuring expected behavior and function calls, spies contribute to test stability, and highly reduces the likelihood of flakiness in our test.

Fake ObjectsFakes do have a working implementation but usually take shortcuts which makes them unsuitable for production. They work and give results like they are the real thing, but take shortcuts to be easier to use in tests.

Let’s take for example, using an in-memory database instead of a real one speeds up tests but wouldn’t work for a live app because it’s too simple or not robust enough. Unlike spies, fakes are more about simplifying things by acting as a lightweight version of the real deal, making tests easier to manage.

Lets look at a simple implementation of Fakes:

// Fake Example with Jestclass InventoryServiceFake { checkStock(itemId) { return Promise.resolve(true); }}test("fake test", () => { const inventoryServiceFake = new InventoryServiceFake(); return inventoryCheck("item123", inventoryServiceFake).then((isInStock) => { expect(isInStock).toBe(true); });}); In the example above, we define a simple implementation of the InventoryService class called InventoryServiceFake. This fake implementation provides a simplified version of the checkStock method, which always resolves to true.

By using a fake object with a constant predictable behavior, we eliminate dependencies on external systems that might introduce flakiness into our test. The test then becomes more deterministic as it always expects true from the fake service. Is it a good practice? Of course it is.

Mock ObjectsMocks in tests act like parts of your code that expect certain actions to happen. They’re set up to watch for specific methods or functions to be called in a certain way. Instead of returning data like some other test doubles, mocks check if the right interactions occur. This means you’re testing how different parts of your code talk to each other, making sure everything is connected correctly.

Let’s look at a simple imlementaton of mocks in Jest. Imagine we have a class that retrieves user data from our API. This class utilizes Axios to make the API call and then returns the data attribute, which encompasses all the users. This could potentially be flaky because it depends on an external network request via Axios.

import axios from "axios";class Users { static async all() { try { const response = await axios.get("https://api.example.com/users"); return response.data.users; } catch (error) { console.error("Error fetching users:", error); return []; } }}export default Users; Now in other to test this method to completely isolate it’s dependence on external factors, We will use jest.mock(‘axios’) to mock the entire axios module.

We then use axios.get.mockResolvedValue(resp) to mock the get method of Axios to resolve with a mock response. The test asserts that the fetched users match the expected users. Thereby making it less prone to flakiness.

import Users from "./Users";import axios from "axios";jest.mock("axios");describe("Users", () => { test("fetching users should return expected data", async () => { const users = [{ name: "Alice" }]; const resp = { data: { users } }; axios.get.mockResolvedValue(resp); const fetchedUsers = await Users.all(); expect(fetchedUsers).toEqual(users); });}); Stub ObjectsStubs in tests can be likened to actors that follow a script. You tell them what to say (or how to respond) when they’re asked something specific during a test. They don’t do anything unexpected; they just give the responses you’ve set up prior to the response or in advance.

This helps make your tests much more predictable by ensuring that parts of your code that are not being tested don’t cause any unexpected results. Fakes are close to stubs but are often used instead of stubs because of their simplicity. The stub example will be seen as the article goes on.

Implementing Test DoublesIn my opinion, there are four ways we should implement test doubles to get the best results. We could start up by:

1. Identifying tests that are flaky due to external dependencies or complex setups.
2. Being able to choose the most appropriate test doubles for the job based on the nature of the test and the dependency.
3. Replace the external dependency with the chosen test double.
4. Refactor the test to use the test double, focusing on clarity and ensuring that the test accurately reflects the intended behavior, and after implementing the test doubles monitor the test for improvements and make adjustments if necessary.

Problems and Test Double SolutionsIn this section, we will be looking at four theoretical Test problems and Test double solutions.

1. Isolation from External DependenciesProblem: Tests that interact with external systems that is databases, and APIs can fail. This is due to unpredictable issues that are outside the control of the test environment. It could be a network issue or an external service downtime.

Solution: Stubs and Fakes can simulate these external systems, ensuring that tests run consistently without being affected by external factors.

1. Controlled BehaviorProblem: One of the causes of Flakiness in tests is dynamic data. Dynamic data could change between test runs.

Solution: Mocks and stubs can be programmed with a fixed response, ensuring that tests receive the same input and follow the same execution path every time, eliminating different results in tests.

1. Time-based FlakinessProblem: Tests relying on real-time clocks or waiting for certain time-based events can fail if the timing isn’t precise on every run. Factors like system load, network delays, or scheduling differences can cause tests to fail even if the code itself is correct.

Solution: Using test doubles like stubs and spies to simulate time or events allows tests to run consistently regardless of real-world timing issues.

1. Resource-intensive testsProblem: Some tests may be flaky because they exhaust system resources (memory, CPU) when running under certain conditions.

Solution: Fakes can be employed to provide lightweight implementations of heavy dependencies, reducing the resource load and avoiding conditions that lead to flakiness.

This is a good way to solve resource intensiveness but it is exposed to limitations, and here is why. If the flakiness shoots from internal resource consumption within your own code, unfortunately, fakes won’t directly solve the problem! You will then need to optimize your code’s resource usage.

Practical Application Of Test Doubles to Ehance Test StabilityIn this section, we will briefly look at a practical implementation of a Test double that could enhance Test stability thereby managing Flakiness in our test:

Problem: Network Dependency and External Service ReliabilityTo address the problem with network dependency and external service reliability, we’ll use a stub. Let’s take for example a function called inventoryCheck(). This function is designed to check if an item is in stock by using an external inventory service which could be flaky. This is what the example looks like:

function inventoryCheck(itemId, inventoryService) { return inventoryService .checkStock(itemId) .then((isInStock) => { return isInStock; }) .catch((error) => { console.error("Error checking inventory:", error); return false; });} Let’s go ahead and employ a stub:

class InventoryServiceStub { checkStock(itemId) { return new Promise((resolve) => { resolve(true); }); }}// Test using the stubtest("inventoryCheck with stub should return true for any item", () => { const inventoryServiceStub = new InventoryServiceStub(); return inventoryCheck("item123", inventoryServiceStub).then((isInStock) => { expect(isInStock).toBe(true); });}); In the code above our test is written with the Jest Framework. The class is a stub for the real inventory service. It simulates the behavior of an inventory service but in a simplified manner.

The checkStock() method in this stub doesn’t perform any real inventory check. Instead, it returns a promise that resolves with true, indicating that any item passed to it is considered to be in stock. This stub is used in testing to isolate the inventoryCheck function from external dependencies, allowing for a controlled environment to test the function’s logic and thereby manage Flakiness in our test.

Thoughts To NoteWill test doubles help solve flakiness in your test? The answer is yes, but while test doubles are a valuable tool for managing flaky tests;

• They should be used simultaneously with techniques like Identifying the core root cause of flakiness (which could be network issues, external system bugs), and Refactoring code to be more deterministic and less dependent on external factors.
• Misusing test doubles can potentially lead to cascading failures in a couple of ways. If the doubles don’t accurately mimic the behavior of the components they’re replacing, tests might pass erroneously, hiding real issues that could cause failures in production.
• Over-reliance on test doubles is not advised. It’s crucial to balance the use of test doubles with integration and end-to-end tests to ease these risks.
• Test doubles won’t detect API contract changes, so ensure you have separate integration tests that interact with the real API.

Closing ThoughtsTest doubles to a great extent helps, in fact if used properly would solve a mass majority of flakiness, but also understanding the simple cause of test fails could go a long way. Thank you for staying this long, I really appreciate your time. Let’s keep testing!

The post Enhancing Test Stability With Test Doubles for Flaky Test Management appeared first on Semaphore.

View Details

IntroductionThis tutorial guides you through various strategies to optimize CI/CD workflows in Semaphore, though many of these practices are also applicable to other CI/CD tools.

Our focus here is on techniques that minimize resource consumption and make your workflows run faster. By the end of this guide, you will understand how to improve feedback loops and project performance through dependency caching, fail-fast strategies, auto-cancel policies, and conditional job executions.

PrerequisitesBefore starting, you should have:

• An active Semaphore account.
• A foundational understanding of CI/CD principles.

Reducing Build Times with Dependency CachingChallenge: A considerable portion of build time in CI/CD pipelines is often consumed by reinstalling dependencies that rarely change, which delays feedback and uses resources unnecessarily.

Solution: Adding Dependency Caching

Two commands, cache store and cache restore can be used to manage dependency caching.

Example: Implementing Caching in a React Project

```

semaphore.yml snippet for caching npm dependenciesblocks: - name: "Install dependencies" task: prologue: commands: - checkout - cache restore $SEMAPHORE_GIT_BRANCH-$(checksum package.json), $SEMAPHORE_GIT_BRANCH, cache-master jobs: - name: npm install commands: - npm install - cache store $SEMAPHORE_GIT_BRANCH-$(checksum package.json) node_modules

`` This configuration checks for changes inpackage.json` and uses the cache when there are no updates, reducing unnecessary installations.

Learn more about caching and its strategies in the documentation.

Using Fail-Fast Strategies to Conserve ResourcesChallenge: Continued execution of jobs after a failure in one job can lead to wasted computational resources.

Solution: Implementing Fail-Fast

Semaphore includes options to immediately stop executing other jobs when a failure is detected, which helps to preserve resources.

Example: Implementing Fail-Fast in your Workflow

```

semaphore.yml snippet for fail-fast strategyversion: v1.0name: Example Semaphore CI pipelineagent: machine: type: e1-standard-2 os_image: ubuntu2204fail_fast: stop: when: "any"

``` This configuration cancels all pending jobs if any job fails.

We can also configure this by using the Workflow Builder:

Learn more about fail fast and its strategies in the documentation.

Canceling Redundant BuildsChallenge: Multiple pipelines running for several commits on the same branch can lead to redundant builds.

Solution: Setting Up Auto-Cancel

The auto-cancel feature helps manage resources by canceling redundant pipeline executions.

Example: Auto Cancel all new commits on existing branch

```

semaphore.yml snippet for auto-cancel policyauto_cancel: queued: when: "branch_has_new_commit"

``` This setting cancels all queued jobs on a branch when a new commit is pushed.

We can also configure this by clicking on the pipeline and going into the Auto-Cancel tab using the Workflow Builder:

Learn more about auto cancel and its strategies in the documentation.

Optimizing Execution with Conditional Job RunsChallenge: Executing jobs when no relevant code changes have occurred increases build times and resource use.

Solution: Using Conditional Job Execution

The change_in function allows jobs to run only when specific changes are detected in the codebase.

Example: Run Tests Block only if folder tests has changed

```

semaphore.yml snippet for conditional job executionblocks: - name: "Run Tests" run: when: "change_in('/tests/')"

`` This configuration runs the test block only if there are changes in thetests` directory.

We can also configure this using the Workflow Builder, by selecting a block and defining the Skip/Run conditions:

Learn more about the change_in command and its strategies in the documentation.

VideoCheck out the accompanying video to see these optimizations in action

ConclusionBy implementing the strategies detailed in this guide, you can decrease build times and operational costs in Semaphore. The techniques provided in this guide can be adapted to other CI/CD platforms as well.

The post A Guide to CI/CD Costs Optimization appeared first on Semaphore.

View Details

Today we are introducing new machine types, Apple Silicon M2 machines. These machines bring reduction of build and test times for developers working on macOS, iOS or any other platform type supported by Apple Hardware.

Who are these for?The new machines are suitable for all projects that are creating applications for Apple’s Ecosystem and use XCode 15 and above.

New Machines Performance BenchmarkWe’ve conducted benchmarks comparing the new M2 machine types against our existing AMD64 ARM machines using the XCode Benchmark repository.

M2 vs ARM64 build performance on Semaphore.Using the new a2-standard-4 machine type (Apple Silicon M2) can reduce your build times by up to three times.

Apple Silicon M2 machine build on Semaphore versus AMD64.Getting Started with Apple Silicon MachinesTo utilize the M2 Silicon machines in your Semaphore projects, specify the machine type in your agent configuration:

version: 1.0name: Apple Based Pipelineagent: machine: type: a2-standard-4 os_image: macos-xcode15 Or use the Workflow Builder to select the machine type:

Please note that the a2-standard-4 machine type only supports the macos-xcode15 image.

ConclusionThe new Apple Silicon machine types reduce build times for developers building applications for macOS, iOS, or any other platform supported by Apple’s hardware.

If your codebase supports XCode 15, you can switch to new a2-standard-4 machines starting today!

For more information check out the documentation.

Follow our YouTube channel for future videos.

Have a benchmark you want to share with us or just want to talk? Join our Discord !

The post Announcing Apple Silicon Machine Types appeared first on Semaphore.

View Details

Before now, the concept of containerization was left solely for the DevOps engineers. However today, the average developer is expected to have a basic understanding of how to containerize their applications. With the most popular containerization tool being Docker.

With this increasing demand for containers, there are some concepts in Docker that most engineers don’t pay atttention to. One of such concepts involves the management of sensitive data in containers. You can observe this in the way most developers hardcode sensitive data like API keys, database credentials, and other secrets directly into their Dockerfiles or environment variables.

This is bad practice and can lead to security vulnerabilities. This article will guide you through the best practices for managing sensitive data in Docker containers. It will teach you Docker’s built-in secrets management feature and how to use it to secure your applications. By the end of this article, you’ll know how to securely pass your sensitive data to your containers.

PrerequisitesThis article will contain a lot of hands-on examples and practical demonstrations. To follow along, you should have the following:

• Basic understanding of containerization especially Docker.
• A computer with Docker installed. Use the official Docker installation guide to install Docker on your machine.
• Basic understanding of container orchestration concepts (optional, for the Docker Swarm section).
• Docker Compose installed on your machine. You can use the official Docker compose installation guide to install Docker compose.

What are Docker Secrets?Docker secrets is a feature for Docker containers, that allows you to use and store sensitive data in a safe and secure way. These sensitive data could be SSH keys, API keys, database credentials, or any other piece of information that should not be shared with everyone who has access to the container.

By default, data transmitted between Docker containers is not encrypted or protected, so it can potentially be intercepted and read by someone with access to the network. This is why Docker secrets comes into the mix, to provide the security needed for your data.

Why Do you Need Docker Secrets?In a situation where you have one of your secrets, say your TLS certificates stored in plain text within your Dockerfile or in your Docker image. This means that anyone who has access to your Dockerfile or Docker image can easily access this secret. This is why you need Docker secrets.

Coupled with its main function of providing security, there are other reasons why you need Docker secrets. Some of these reasons include:

• Superior Security Over Environment Variables: Docker secrets are generally considered more secure for storing sensitive data compared to environment variables. Docker secrets are stored in a virtual filesystem managed by Docker, and they are only accessible to the containers that have been granted access to them. Environment variables, on the other hand, are visible to anyone with access to the container, including processes running within the container.
• Single Storage Location: Docker secrets are stored in a single location on the Docker host, and they are accessible to all containers running on the host. This makes it easy to manage secrets across multiple containers which is usually the case when using Docker Swarm or Docker Compose.
• Ease Of Management: Although not utilized as often, using Docker secrets isn’t rocket science. It’s quite easy to manage and use Docker secret via the Docker CLI.
• Abstraction Between Containers and Secrets: In a case where you have multiple environments, say, test, local and production. By using Docker secrets, your containers don’t need to be aware of the specific credentials used in each environment. Instead, they rely on a consistent secret name, abstracting away the underlying details. This flexibility allows you to deploy the same container image across multiple environments without modification.

How do Docker Secrets Work?You can only use Docker secrets when working with a collection of containers such as in Docker Swarm or Docker compose. When you add some secrets to a Docker Swarm, Docker will use a mutual TLS connection to securely pass these stored secrets to the Swarm manager. These secrets are stored in something called a Raft log, which is like an encrypted record-keeping system.

The Raft log along with the secrets are then replicated across all the managers in the Swarm. This ensures that even if one manager fails, the secret and other management data remain available.

Figure 1. Flow diagram for Docker secrets architectureWhen you want a service to use a secret, Docker takes the encrypted secret from the Raft log, decrypts it, and mounts it into the container’s memory as a special kind of file system. This file system is only accessible to the service’s containers, and it’s mounted as a read-only file system.

By default, in Linux containers, this mounted secret is placed in /run/secrets/<secret_name>, while in Windows containers, it’s put in C:\ProgramData\Docker\secrets. However, you can choose a different location if needed.

Nodes in the Swarm (the machines running Docker containers) only have access to encrypted secrets if they are Swarm managers or if they are running tasks for services that have been granted access to those secrets.

When a container stops running, any secrets it had access to are removed from the in-memory filesystem of that container and wiped from the node’s memory.

In the case where a node loses connection to the Swarm while running a task with access to a secret, the task can still access the secrets it had. However, it won’t receive updates until it reconnects to the Swarm.

How to Create a Docker SecretDocker provides a set of commands that make it easy to create and manage secrets. By running the docker secret createcommand, you can create a new secret and store it in the Docker host. However, there are two ways you can use this command.

• Using standard input: This is the simplest way to create a secret. The syntax for this is:

echo "mysecretpassword" | docker secret create my_secret -kotjiwncv3kp8uyyj8d7kzesn * You’re creating a secret named my_secret and passing the secret data mysecretpassword through a pipe | from the echo command. Docker will then read the secret data from standard input. * Using a file: You can also create a secret from a file. The syntax for this is:

docker secret create my_secret_file /path/to/my_secret_filel95k6p8h7q6xgji9lxh164ibs In this case, you’re passing the my_secret_file to the docker secret create command. Docker will then read the secret data from the file and create a secret named my_secret_file.

The kotjiwncv3kp8uyyj8d7kzesn and l95k6p8h7q6xgji9lxh164ibs are the IDs of the secrets you just created. You can use these IDs to reference the secrets in other Docker commands.

To view your newly created secret, run the docker secret ls command. You should have a similar output:

ID NAME DRIVER CREATED UPDATEDkotjiwncv3kp8uyyj8d7kzesn my_secret 3 minutes ago 3 minutes agol95k6p8h7q6xgji9lxh164ibs my_secret_file About a minute ago About a minute ago How to Manage Docker SecretsThe docker secret command provides a set of subcommands that allow you to manage secrets. Below are some of the most commonly used subcommands:

• docker secret ls: As seen above, this command lists all the secrets stored on the Docker host.
• docker secret inspect: This command provides detailed information about a specific secret. You can use it to view the metadata of a secret, such as its ID, name, and creation date.