{"slug": "implementing-distributed-tracing-in-a-golang-application", "title": "Implementing Distributed Tracing in a Golang application", "summary": "SigNoz and OpenTelemetry enable distributed tracing in a Golang application with three microservices, using OpenTelemetry Go SDK for instrumentation and OTLP export to SigNoz for visualization. The open-source tools provide insights into complex distributed systems, helping engineering teams identify performance issues across services.", "body_md": "# Implementing Distributed Tracing in a Golang application\n\nIn this article, we will implement distributed tracing for a Golang application with three microservices. To implement distributed tracing, we will be using open-source solutions - SigNoz and OpenTelemetry, so you can easily follow the tutorial.\n\nWhat is distributed tracing?\n\nGo is the one major language with no runtime auto-instrumentation agent: you wire the OpenTelemetry Go SDK into your code, set up a tracer provider, and export spans over [OTLP](https://signoz.io/blog/what-is-otlp/). For lower-touch coverage, eBPF-based auto-instrumentation (the OpenTelemetry Go instrumentation project) is maturing as the agentless option, and either path lands traces in an OTLP backend like SigNoz.\n\nModern application architecture using cloud-native, containerization, and microservices is a very complex distributed system. A typical web-search example will illustrate some of the challenges such a system needs to address.\n\nA front-end service may distribute a web query to many hundreds of query servers. The query may also be sent to a number of other sub-systems that may process advertisements or look for specialized results like images, news, etc. This might involve database access, cache lookup, network call, etc. In total, thousands of machines and many different services might be needed to process one search query.\n\nMoreover, web-search users are sensitive to delays, which can be caused by poor performance in any sub-system. An engineer looking only at the overall latency may know there is a problem but may not be able to guess which service is at fault nor why it is behaving poorly. And such services are also not written and managed by a single team. Also, day by day, new components might get added to the system. Distributed tracing provides insights into the inner workings of such a complex system. Tracing such complex systems enables engineering teams to set up an observability framework.\n\nDistributed tracing gives insights into how a particular service is performing as part of the whole in a distributed software system. It involves passing a [trace context](https://signoz.io/blog/context-propagation-in-distributed-tracing/) with each user request which is then passed across hosts, services, and protocols to track the user request.\n\nIn this article, we will use OpenTelemetry and SigNoz to enable [distributed tracing](https://signoz.io/blog/distributed-tracing/) in a sample Golang application with microservices. But before we deep dive into the implementation steps, let us give you a brief context on OpenTelemetry and SigNoz.\n\nOpenTelemetry and SigNoz\n\n[OpenTelemetry](https://opentelemetry.io/) is a vendor-agnostic set of tools, APIs, and SDKs used to instrument applications to create and manage telemetry data(logs, metrics, and traces). It aims to make telemetry data a built-in feature of cloud-native software applications.\n\nOpenTelemetry provides the instrumentation layer to generate and export your telemetry data to a backend. Then, you need to choose a backend tool that will provide the data storage and visualization for your telemetry data. That’s where SigNoz comes into the picture.\n\n[SigNoz](https://signoz.io/) is a full-stack open-source observability tool that provides logs, metrics, and traces under a single pane of glass.\n\nOpenTelemetry is the way forward for cloud-native application owners who want to set up a robust observability framework. It also provides you the freedom to choose any backend analysis tool. SigNoz is built to support OpenTelemetry natively, thus making a great combo.\n\nDistributed Tracing in a Golang application\n\nWe will demonstrate implementing distributed tracing in a Golang application in the following sections:\n\n- Instrumenting the Golang app with OpenTelemetry\n- Running the sample Golang application\n- Visualizing traces data with SigNoz dashboards\n\nPrerequisites\n\n- Go (version ≥ 1.16)\n- For installation see\n[getting started](https://go.dev/doc/install)\n\n- For installation see\n- MySQL 8\n- Download the MySQL community version from\n[here](https://dev.mysql.com/downloads/mysql/) - If your MySQL is configured with a password, update it here:\n[https://github.com/SigNoz/distributed-tracing-golang-sample/blob/master/.env](https://github.com/SigNoz/distributed-tracing-golang-sample/blob/master/.env)\n\n- Download the MySQL community version from\n`serve`\n\nfor frontend. For installation see:[https://www.npmjs.com/package/serve](https://www.npmjs.com/package/serve)- SigNoz - For instructions, please refer to\n[Installing SigNoz](#installing-signoz)section.\n\nInstalling SigNoz\n\nFirst, you need to install SigNoz so that OpenTelemetry can send the data to it.\n\nSigNoz can be installed on macOS or Linux computers in just three steps by using a simple installation script.\n\nThe install script automatically installs Docker Engine on Linux. However, on macOS, you must manually install [Docker Engine](https://docs.docker.com/engine/install/) before running the install script.\n\n```\ngit clone -b main https://github.com/SigNoz/signoz.git\ncd signoz/deploy/\n./install.sh\n```\n\nYou can visit our documentation for instructions on how to install SigNoz using Docker Swarm and Helm Charts.\n\nWhen you are done installing SigNoz, you can access the UI at `http://localhost:3301/application`\n\n.\n\nInstrumenting the Golang app with OpenTelemetry\n\nWe have built a sample Golang application for the purpose of this tutorial. It has 3 services:\n\n- user-service\n- payment-service, and\n- order-service\n\nThese services are instrumented with OpenTelemetry libraries, and when they interact with each other, OpenTelemetry emits the telemetry data to [OTel collector](https://signoz.io/blog/opentelemetry-collector-complete-guide/) which comes bundled with SigNoz.\n\n**Step 1:** **Clone sample Golang app repository and go to the root folder**\nWe will be using a sample Golang app at this [GitHub repo](https://github.com/SigNoz/distributed-tracing-golang-sample).\n\n```\ngit clone https://github.com/SigNoz/distributed-tracing-golang-sample.git\ncd distributed-tracing-golang-sample\n```\n\n**Step 2: Install the required dependencies**\n\nCheck the list of all the required modules from go.mod. For OpenTelemetry, we need:\n\n```\ngo.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux v0.32.0\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.32.0\ngo.opentelemetry.io/otel v1.7.0\ngo.opentelemetry.io/otel/exporters/otlp/otlptrace v1.7.0\ngo.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.7.0\ngo.opentelemetry.io/otel/sdk v1.7.0\ngo.opentelemetry.io/otel/trace v1.7.0\ngithub.com/XSAM/otelsql v0.14.1\n```\n\nAll the dependencies can be installed using:\n\n```\ngo mod tidy\ngo mod vendor\n```\n\n**Step 3: Configure the OpenTelemetry collector**\n\nIdeally, you should start OpenTelemetry at the beginning of main, before any other services start running. When your program exits, call `Shutdown`\n\non the SDK to ensure the last bit of telemetry is flushed before the program exits.\n\n```\ntp := config.Init(serviceName)\ndefer func() {\n\tif err := tp.Shutdown(context.Background()); err != nil {\n\t\tlog.Printf(\"Error shutting down tracer provider: %v\", err)\n\t}\n}()\n// tracer is later used to create spans\ntracer = otel.Tracer(serviceName)\n```\n\nWe also initialized the `tracer`\n\nwhich is later used to create custom spans.\n\nLet’s now understand what does `Init`\n\nfunction in `config/config.go`\n\ndoes.\n\n**Initialize exporter:**\n\nThe exporter in SDK is responsible for exporting the telemetry signal (trace) out of the application to a remote backend, logging to a file, etc. In this demo, we are creating a gRPC exporter to send out traces to an OpenTelemetry Collector backend running at collectorURL (SigNoz). It also supports TLS and application auth using headers.\n\n```\nsecureOption := otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, \"\")) // config can be passed to configure TLS\nif len(insecure) > 0 {\n\tsecureOption = otlptracegrpc.WithInsecure()\n}\nexporter, err := otlptrace.New(\n\tcontext.Background(),\n\totlptracegrpc.NewClient(\n\t\tsecureOption,\n\t\totlptracegrpc.WithEndpoint(collectorURL),\n\t\totlptracegrpc.WithHeaders(headers),\n\t),\n)\n```\n\n**Construct trace provider:**\n\nTracerProvider provides access to instrumentation Tracers. We configure it to sample all the traces and send the traces in batches to the collector. The resource describes the object that generated the telemetry signals. Essentially, it must be the name of the service or application. We set it to`serviceName`\n\n:\n\n```\ntraceProvider := sdktrace.NewTracerProvider(\n\tsdktrace.WithSampler(sdktrace.AlwaysSample()),\n\tsdktrace.WithSpanProcessor(sdktrace.NewBatchSpanProcessor(exporter)),\n\tsdktrace.WithResource(resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceNameKey.String(serviceName))),\n)\n```\n\nNow, we are ready to configure various components in our application.\n\n**Step 4:** **Instrument HTTP handler with OpenTelemetry**\n\nWe are using `gorilla/mux`\n\nfor the HTTP router. It can be instrumented with OpenTelemetry using [otelmux](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux):\n\n```\nrouter.Use(otelmux.Middleware(serviceName))\n```\n\nNow, all the HTTP calls pass through the OpenTelemetry middleware.\n\nOur services communicate with each other using HTTP APIs. We need to configure our client to pass on the tracing metadata. We can do that using:\n\n```\nfunc SendRequest(ctx context.Context, method string, url string, data []byte) (*http.Response, error) {\n\trequest, err := http.NewRequestWithContext(ctx, method, url, bytes.NewBuffer(data))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"create request error: %w\", err)\n\t}\n\n\tclient := http.Client{\n\t\t// Wrap the Transport with one that starts a span and injects the span context\n\t\t// into the outbound request headers.\n\t\tTransport: otelhttp.NewTransport(http.DefaultTransport),\n\t\tTimeout:   10 * time.Second,\n\t}\n\n\treturn client.Do(request)\n}\n```\n\nThe `ctx`\n\nparameter contains the tracing metadata of the parent span. So now the client sends the metadata and the server can extract this and connect the tracing information of various services.\n\nApart from the instrumentation already provided by [otelhttp](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp) library, we might want to have custom spans for various purposes (e.g. trace database call, or trace function execution). We can do that using:\n\n```\nctx, span := tracer.Start(r.Context(), \"update user amount\")\ndefer span.End()\n```\n\nWe can also attach attributes, events, etc. to this span. Please refer to the [documentation](https://pkg.go.dev/go.opentelemetry.io/otel/trace#Span) for that.\n\n**Step 5:** **Instrument MySQL with OpenTelemetry**\n\nDatabase lies in the hot path for most of the applications and any insights into its performance are valuable. We instrument it with the help of [github.com/XSAM/otelsql](http://github.com/XSAM/otelsql). And while making any DB call, we pass on the context.\n\n```\ndb, err = otelsql.Open(\"mysql\", datasourceName(username, password, host, dbName), otelsql.WithAttributes(\n\t\tsemconv.DBSystemMySQL,\n\t))\n....\n....\nres, err := stmt.ExecContext(ctx, p.Vars...)\n```\n\n[http://github.com/XSAM/otelsql](http://github.com/XSAM/otelsql) is not yet officially supported by OpenTelemetry.\n\nRunning the sample Golang application\n\n**Step 1: Configuration**\n\nTo set up OpenTelemetry to collect and export telemetry data, you need to specify OTLP (OpenTelemetry Protocol) endpoint. It consists of the IP of the machine where SigNoz is installed and the port number at which SigNoz listens.\n\nOTLP endpoint for SigNoz - `<IP of the machine>:4317`\n\nIf you have installed SigNoz on your local machine, then your endpoint is `127.0.0.1:4317`\n\n.\n\nIf you have installed SigNoz on some domain, then your endpoint is `test.com:4317`\n\nDo not use `http`\n\nor `https`\n\nin the IP address.\n\nConfiguration for the following can be set up in `.env`\n\n```\n# service config\nUSER_URL=localhost:8080\nPAYMENT_URL=localhost:8081\nORDER_URL=localhost:8082\n\n# database config\nSQL_USER=root\nSQL_PASSWORD=password\nSQL_HOST=localhost:3306\nSQL_DB=signoz\n\n# telemetry config\nOTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317\nINSECURE_MODE=true\n```\n\n**Step 2: Run the microservices**\n\nAs we have already cloned the repo in the above section, from the root folder, run these commands, each in a separate terminal:\n\n```\ngo run ./users\ngo run ./payment\ngo run ./order\n```\n\n**Step 3: Confirm table creation:**\n\nAfter running the services, check if the tables `ORDERS`\n\nand `USERS`\n\nare created using the commands below:\n\n```\nmysql> use signoz;\nmysql> show tables;\n```\n\nVisualizing Distributed Tracing data with Signoz\n\nTo visualize the trace data with SigNoz, we first need to generate some user data by interacting with the frontend.\n\n**Generating user data by interacting with the sample app:**\n\nYou need to generate some user data to see how it appears in the SigNoz dashboard. The sample application comes with a UI to interact with the app. Use the below command in the root folder to launch the UI:\n\n```\nserve -l 5000 frontend\n```\n\nNow go to the app frontend running at `http://localhost:5000`\n\n. Perform the below given steps 4-5 times to generate some data.\n\n-\nCreate a user:\n\nClick`Create User`\n\nbutton to create a new user in the MySQL db. -\nTransfer the fund:\n\nTransfer some amount by clicking`Transfer Fund`\n\nbutton. -\nPlace an order:\n\nPlace an order by selecting a product from the dropdown.\n\nNow go to the SigNoz dashboard (running on `http://localhost:3301/`\n\nby default), wait for some time, and refresh the dashboard. You will notice the list of service names that we configured:\n\n- user-service\n- order-service\n- payment-service\n\nAnalyze traces and metrics using the Signoz dashboard\n\nIn the metrics tab, you can see Application Metrics, External Calls, and Database Calls:\n\n**Application metrics:**\n\nHere, we can see the application latency, requests per second(rps), error percentage, and the endpoints that were hit for a given service.\n\n**External Calls:**\n\nHere, you can see the metrics about the calls made to external services. In our case, we are running the services on localhost; hence we see a single line. Metrics like external call duration (by address) give a quick glimpse of the network connectivity with the external service. This might be useful to detect a network issue.\n\nFor more features on metrics, please read the [documentation](https://signoz.io/docs/userguide/metrics/).\n\n**Identify latency issues with Flamegraphs and Gantt charts**\n\nYou can inspect each event in the spans table with Flamegraphs and Gantt charts to see a complete breakdown of the request. Establishing a sequential flow of the user request along with info on time taken taken by each part of the request can help identify latency issues quickly. Let’s see how it works in the case of our sample Go app.\n\nGo to the service name filter on the left and select `order-service`\n\n. Now select any span:\n\nHere, expanding on `insert order`\n\nyou will see the time utilised in various SQL DB calls.\n\nAlso, note that we also have additional information about the query that was run in the tags panel on the right.\n\nConclusion\n\nDistributed tracing is a powerful and critical toolkit for developers creating applications based on microservices architecture. For Golang applications using microservices architecture, distributed tracing can enable a central overview of how requests are performing across microservices.\n\nThis lets application owners reconstruct the whole path of the request and see how individual components performed as part of the entire user request.\n\nOpenTelemetry and SigNoz provide a great open-source solution to implement distributed tracing for your applications. You can try out SigNoz by visiting its GitHub repo 👇\n\nIf you are someone who understands more from video, then you can watch the below video tutorial on the same with SigNoz.\n\nIf you have any questions or need any help in setting things up, join our slack community and ping us in `#support`\n\nchannel\n\n**Further reading**\n\nTo go deeper, start with the fundamentals of [OpenTelemetry tracing](https://signoz.io/blog/opentelemetry-tracing/) and how [spans in distributed tracing](https://signoz.io/blog/distributed-tracing-span/) capture each unit of work in a request. When you're ready to pick a backend, compare the leading [distributed tracing tools](https://signoz.io/blog/distributed-tracing-tools/) and see how tracing relates to broader monitoring in [APM vs distributed tracing](https://signoz.io/blog/apm-vs-distributed-tracing/).\n\nFor applying these ideas in practice, our guide to [microservices observability](https://signoz.io/blog/microservices-observability-with-distributed-tracing/) shows how tracing fits into a full observability setup, and if you work across stacks, see the same walkthrough for [distributed tracing in Java](https://signoz.io/blog/distributed-tracing-java/).\n\nFurther, you can understand more about SigNoz by going through\n[SigNoz - an open-source alternative to DataDog](https://signoz.io/blog/open-source-datadog-alternative/).", "url": "https://wpnews.pro/news/implementing-distributed-tracing-in-a-golang-application", "canonical_source": "https://signoz.io/blog/distributed-tracing-golang", "published_at": "2026-07-06 00:00:00+00:00", "updated_at": "2026-07-07 06:30:27.292708+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["SigNoz", "OpenTelemetry", "Golang", "OTLP"], "alternates": {"html": "https://wpnews.pro/news/implementing-distributed-tracing-in-a-golang-application", "markdown": "https://wpnews.pro/news/implementing-distributed-tracing-in-a-golang-application.md", "text": "https://wpnews.pro/news/implementing-distributed-tracing-in-a-golang-application.txt", "jsonld": "https://wpnews.pro/news/implementing-distributed-tracing-in-a-golang-application.jsonld"}}