Basicserversetupadwance

Basic Server Setup Guide

Purpose: Prepare a Linux server (Amazon Linux or Ubuntu) with hostname, system packages, timezone, utilities, database clients, AWS CLI, and logging setup for applications.

1. Set Hostname

Amazon Linux / Ubuntu:

sudo hostnamectl set-hostname <your-hostname>

2. Update System Packages

Amazon Linux:

sudo yum update -y

Ubuntu:

sudo apt update && sudo apt upgrade -y

3. Set Timezone to IST

Amazon Linux / Ubuntu:

sudo timedatectl set-timezone Asia/Kolkata

4. Install Basic Utilities

Amazon Linux:

sudo yum install -y wget curl git unzip zip tar net-tools bind-utils htop tree vim

Ubuntu:

sudo apt install -y wget curl git unzip zip tar net-tools dnsutils htop tree vim

5. Enable SSH Key Authentication (for Jenkins or root access)

mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "<your-public-key>" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

6. Install Database Clients

Amazon Linux:

# MySQL client
sudo yum install -y mariadb
 
# PostgreSQL client
sudo amazon-linux-extras enable postgresql14
sudo yum install -y postgresql
 
# Verify
mysql --version
psql --version

Ubuntu:

# MySQL client
sudo apt install -y mysql-client
 
# PostgreSQL client
sudo apt install -y postgresql-client
 
# Verify
mysql --version
psql --version

7. Install AWS CLI (v2 recommended)

Amazon Linux / Ubuntu:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

8. Set Up Systemd Service for Applications

Purpose: Run Python or Go apps as systemd services with persistent logging and automatic startup.

8.1 Create Service Unit File

Example: Python (FastAPI with uvicorn)

[Unit]
Description=<app-name> FastAPI Application
After=network.target
 
[Service]
User=root
Group=root
WorkingDirectory=/home/<app-name>
ExecStart=/home/<app-name>/venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=5
StandardOutput=append:/var/log/<app-name>/app.log
StandardError=append:/var/log/<app-name>/app.log
 
[Install]
WantedBy=multi-user.target

Example: Go Application

[Unit]
Description=<app-name> Go Application
After=network.target
 
[Service]
User=root
Group=root
WorkingDirectory=/home/<app-name>
ExecStart=/home/<app-name>/<app-name>-service
Restart=always
RestartSec=5
StandardOutput=append:/var/log/<app-name>/app.log
StandardError=append:/var/log/<app-name>/app.log
 
[Install]
WantedBy=multi-user.target

8.2 Register & Start Service

sudo systemctl daemon-reload
sudo systemctl enable <app-name>
sudo systemctl start <app-name>
sudo systemctl status <app-name>

8.3 Set Up Logging Directory

sudo mkdir -p /var/log/<app-name>
sudo chown root:root /var/log/<app-name>

8.4 Configure Log Rotation

Create /etc/logrotate.d/<app-name>:

/var/log/<app-name>/app.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    dateext
    dateformat -%Y-%m-%d
}

Note on copytruncate: Ensures Fluent Bit (or any tailing agent) continues reading logs without interruption.

8.5 Force Log Rotation (Testing)

sudo logrotate -f /etc/logrotate.d/<app-name>
ls -lh /var/log/<app-name>/

app.log → fresh, current log
app.log-YYYY-MM-DD.gz → rotated & compressed

8.6 View Logs

tail -f /var/log/<app-name>/app.log

✅ Logs are written to /var/log/<app-name>/app.log, rotated daily, and services start automatically on reboot.

9. Set Up Fluent Bit for Log Collection (to Kibana / S3)

Install Fluent Bit:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Configure /etc/fluent-bit/fluent-bit.conf (example):

[SERVICE]
    Flush        60
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
    storage.path /var/lib/fluent-bit/storage
    storage.sync normal
    storage.backlog.mem_limit 128M
 
[INPUT]
    Name         systemd
    Tag          <app-name>
    Systemd_Filter
    Read_From_Tail On
    DB           /var/lib/fluent-bit/systemd-<app-name>.db
    _SYSTEMD_UNIT=<app-name>.service
 
[FILTER]
    Name   modify
    Match  <app-name>
    Rename MESSAGE log
    Remove_regex ^(?!log$).*
 
[OUTPUT]  # S3
    Name        s3
    Match       <app-name>
    bucket      example-co-in-aws-application-logs
    region      ap-south-1
    total_file_size 50M
    upload_timeout 1m
    store_dir   /var/lib/fluent-bit/s3
    s3_key_format /application-logs/<app-name>-prod/$TAG/%Y/%m/%d/%H/<app-name>-$UUID.json
    upload_chunk_size 5M
    use_put_object On
    content_type application/json
 
[OUTPUT]  # Logstash
    Name   forward
    Match  <app-name>
    Host   <logstash-hostname-or-ip>
    Port   5044
    # tls settings if needed
    # tls On
    # tls.verify Off
    # tls.ca_file /etc/ssl/certs/ca-bundle.crt

10. Set Up Logstash to Receive Logs

Example /etc/logstash/conf.d/<app-name>.conf:

input {
    tcp {
        port 5001 # Use available port
        codec json
        tags ["app-name"]
    }
}
 
output {
    if "app-name" in [tags] {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            user => "elastic"
            password => "your-password"
            ssl_certificate_verification => false
            index => "app-name-%{+YYYY.MM.dd}"
        }
    }
}

Important Notes:

Logstash domain usually works internally in VPC (logstash.internal.example.co.in).
If Kibana is in another VPC, allow ports (e.g., 5003) via VPC peering.

✅ Outcome:

Server is fully configured with hostname, packages, timezone, utilities, database clients, AWS CLI.
Applications run as systemd services with persistent logging.
Fluent Bit collects logs and forwards them to S3 and Logstash/Kibana reliably.

Baseserversetup Dailycmd