Route cloud app logs into a pipeline

This guide shows you how to simulate, send, and verify a single JSON log event within an observability pipeline workflow.

Goal: send a mock app log, verify pipeline ingestion, and prepare it for routing to destinations like Splunk, Elasticsearch, or Amazon S3.

Why this matters

Before you build complex parsing rules or route data to expensive security information and event management (SIEM) tools, you must verify your ingestion path. This workflow confirms that:

1. Your API token works.
2. The pipeline accepts your JSON structure.
3. You can see the data in real time.

Before you start

Make sure you have:

• An API token with ingest permissions.
• Your unique stream ID from the platform UI.
• Terminal access to run curl, or Postman.

Pipeline overview

Mermaid (image)

Mermaid (code)

flowchart LR
    A["Cloud App"] -->|JSON Log| B["Ingest API"]
    B --> C{"Routing Engine"}
    C -->|Match Rule| D["Live Data Viewer"]
    C -->|No Match| E["Drop / Default Bucket"]

ASCII

ASCII diagram

  [Cloud App]
       |
       | |JSON Log|
       v
  [Ingest API]
       |
       v
{ Routing Engine }
       |
       +-----------------------+
       |                       |
       v |Match Rule|          v |No Match|
+-----------------------+    +-----------------------+
|   Live Data Viewer    |    | Drop / Default Bucket |
+-----------------------+    +-----------------------+

1. Create or select a stream

In your observability platform:

1. Go to Streams > Management.
2. Click New Stream or select default-logs.
3. Copy the Stream ID, for example, st_12345.
4. (Optional) Set a retention policy. The default is usually 7 days.

2. Create a sample log event

Simulate a cloud app log by saving the following as app-log.json.

app-log.json

{
  "timestamp": "2025-11-15T08:30:00Z",
  "service": "payment-gateway",
  "level": "ERROR",
  "message": "Transaction failed: Gateway timeout",
  "transaction_id": "txn_998877",
  "meta": {
    "region": "us-east-1",
    "customer_id": "cus_554433"
  }
}

note

Keep the timestamp in International Organization for Standardization (ISO) 8601 format to ensure proper indexing.

3. Send the log event

Use curl to simulate the app sending the log.

cURL

export STREAM_ID="your_stream_id_here"
export API_TOKEN="your_api_token_here"

curl -X POST "https://api.observability-platform.com/v1/streams/ingest" \
     -H "Authorization: Bearer $API_TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Stream-Id: $STREAM_ID" \
     -d @app-log.json

Expected response

Response

{
  "status": "accepted",
  "ingest_id": "evt_abc12345",
  "accepted_items": 1
}

If you see status: accepted (HTTP 202 Accepted), the pipeline received the data.

4. Verify your log in the live data viewer

1. Navigate to Explore or Live Tail in the platform UI.
2. Select your stream (st_12345).
3. Add a filter: service:payment-gateway.
4. Your event should appear instantly:

Live tail output

[ERROR] 2025-11-15T08:30:00Z service=payment-gateway msg="Transaction failed..."

5. Route logs to a destination

Now that data is flowing, create a routing rule:

1. Go to Pipelines > Routing Rules.
2. Click New Rule.
3. Filter: level:ERROR
4. Action: route to S3-Archive and Slack-Alerts.
5. Save & Deploy.

Send the curl request again.

• Check your S3 bucket.
• Check your Slack channel.

6. Optimize the pipeline

Now that you have a working flow, consider adding:

• Remapping: remap meta.customer_id to a top-level attribute for easier filtering and faceting.
• Parsing: extract error codes or transaction IDs from raw message strings using a Grok parser.
• Enrichment: add a team_owner tag based on the service name.
• Masking: obfuscate customer_id if it's personally identifiable information (PII).

Troubleshooting

Issue / Status Code	Potential Cause / Resolution
401 Unauthorized	Check your API token permissions.
400 Bad Request	Verify your JSON syntax.
404 Not Found	Verify the X-Stream-Id is correct.
Timestamp issues	Ensure you're using Coordinated Universal Time (UTC) ISO 8601 format.
Data not showing	Check if you have an active filter that excludes your new log.

Next steps

• Log Ingestion API
• Event streams and observability pipelines
• Parallel observability overview
• Create a Postman mock server
• Try sending different log types, such as VM logs, container logs, and API events.