You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Scope: Plan for deployments in case of no internet access #69
We need Internet access for various components when installing BHASAI - https://github.com/BharatSahAIyak/devops (specially pulling Docker Images). Figure out all the components which require internet access and create a process for deploying without Internet Access.
Components
System Packages
Docker Images
Domain names used. E.g. In caddy's docker-compose.yaml
Questions: How is the ssh connection b/w no internet sever and other server being established.
Found this: VPN: Set up a VPN on another server within the same network as the isolated server. Once connected to the VPN, you can SSH into the isolated server using its local IP address.
changed the title [-]Plan for deployments in case of no internet access[/-][+]Scope: Plan for deployments in case of no internet access[/+]on Aug 23, 2024
To install build-essential on a server without internet access, you'll need to download the necessary packages on a machine with internet access and then transfer them to the server. Here's how you can do it:
Step 1: Download the Packages on a Machine with Internet Access
On a machine with internet access, update your package list:
sudo apt-get update
Download the build-essential package and its dependencies using apt-get with the --download-only option:
SSH into your server and navigate to the destination directory:
ssh user@server_ip
cd /path/to/destination
Extract the compressed file:
tar -xzvf build-essential-packages.tar.gz
Step 3: Install the Packages on the Server
Navigate to the directory containing the .deb files:
cd build-essential-packages
Install all the packages using dpkg:
sudo dpkg -i *.deb
If there are any missing dependencies, fix them by running:
sudo apt-get install -f
If your system does not have internet access and you run apt-get install -f, it might not be able to fix the broken dependencies unless you have already downloaded the necessary packages and dependencies to your local cache or have them available in a local repository. In such cases, you would need to manually download the required packages and install them.
This will install any missing dependencies from the .deb files that are already present in the directory.
Summary
Download: Use a machine with internet access to download build-essential and its dependencies.
Transfer: Move the files to the server using scp.
Install: Use dpkg on the server to install the packages.
This method allows you to install build-essential without direct internet access on the server.
To build a Docker image without internet access, especially when your Dockerfile references external base images like hashicorp/consul-template:0.37.2 and vault:1.13.3, you need to ensure that these images are available locally on the machine where the build will occur. Here’s how you can do this:
Steps to Build the Image Without Internet
Pre-download the Required Images:
On a machine with internet access, pull the required Docker images:
For setting up docker :
Here are the commands for downloading Docker packages on a Mac, transferring them to an Ubuntu machine using scp, and then installing them on the Ubuntu machine.
We can use any method that can utilse the ssh access that we have. Does gdown utilise ssh? As per my understanding it needs internet
Also, can you share total amount of data in GB (split modules wise - e.g., for docker, for gpu drivers, etc) we will need to transfer in case of offline deployment and what speed you are getting during scp. We can accordingly plan on timelines in future for offline deployments.
Activity
[-]Plan for deployments in case of no internet access[/-][+]Scope: Plan for deployments in case of no internet access[/+]GJS2162 commentedon Aug 26, 2024
BHASAI is this https://github.com/BharatSahAIyak/docker-bhasai, right?
singhalkarun commentedon Aug 26, 2024
https://github.com/BharatSahAIyak/devops (added now in issue description also.
GJS2162 commentedon Aug 26, 2024
Exploring Tasks :
curl -I https://www.google.com/sudo iptables -D OUTPUT -p tcp --dport 80 -o enp0s1 -j DROPsudo iptables -D OUTPUT -p tcp --dport 443 -o enp0s1 -j DROPsudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64scp yq_linux_amd64 barman@192.168.64.12:/tmp/yq_linux_amd64sudo mv /tmp/yq_linux_amd64 /usr/bin/yq sudo chmod +x /usr/bin/yqGJS2162 commentedon Aug 27, 2024
To install
build-essentialon a server without internet access, you'll need to download the necessary packages on a machine with internet access and then transfer them to the server. Here's how you can do it:Step 1: Download the Packages on a Machine with Internet Access
On a machine with internet access, update your package list:
Download the
build-essentialpackage and its dependencies usingapt-getwith the--download-onlyoption:This will download all the necessary
.debfiles to your system's cache (usually in/var/cache/apt/archives).Collect the downloaded
.debfiles:This will copy all the
.debfiles into the~/build-essential-packagesdirectory.Compress the folder containing the
.debfiles to make it easier to transfer:Step 2: Transfer the Packages to the Server
Use
scpto transfer the compressed file to your server (replaceuser@server_ip:/path/to/destinationwith your server details):SSH into your server and navigate to the destination directory:
ssh user@server_ip cd /path/to/destinationExtract the compressed file:
Step 3: Install the Packages on the Server
Navigate to the directory containing the
.debfiles:cd build-essential-packagesInstall all the packages using
dpkg:sudo dpkg -i *.debIf there are any missing dependencies, fix them by running:
If your system does not have internet access and you run apt-get install -f, it might not be able to fix the broken dependencies unless you have already downloaded the necessary packages and dependencies to your local cache or have them available in a local repository. In such cases, you would need to manually download the required packages and install them.
This will install any missing dependencies from the
.debfiles that are already present in the directory.Summary
build-essentialand its dependencies.scp.dpkgon the server to install the packages.This method allows you to install
build-essentialwithout direct internet access on the server.GJS2162 commentedon Aug 27, 2024
To build a Docker image without internet access, especially when your
Dockerfilereferences external base images likehashicorp/consul-template:0.37.2andvault:1.13.3, you need to ensure that these images are available locally on the machine where the build will occur. Here’s how you can do this:Steps to Build the Image Without Internet
Pre-download the Required Images:
Save the Images:
docker savecommand:Transfer the Images to the Server:
scp,rsync, or any other method you prefer:Load the Images on the Server:
docker loadcommand:Build the Docker Image:
hashicorp/consul-template:0.37.2andvault:1.13.3images during the build process.Explanation
docker pull: Downloads the specified image from the Docker Hub (or another configured registry).docker save: Exports the image to a tar file, which can be transferred to another machine.docker load: Imports the image from the tar file into the Docker daemon on the target machine.docker-compose build: Builds the image using the specified Dockerfile, leveraging the loaded images locally without requiring internet access.By following these steps, you can build your Docker image on a server without internet access using the necessary base images.
GJS2162 commentedon Aug 27, 2024
For setting up docker :
Here are the commands for downloading Docker packages on a Mac, transferring them to an Ubuntu machine using
scp, and then installing them on the Ubuntu machine.Step 1: Making the zip for pacakges:
Create a directory to store the packages:
mkdir -p ~/docker-packages**Download Docker Packages on the Ubuntu Machine
tar -czvf docker-packages.tar.gz *.debZip the packages for easy transfer:
tar -czvf docker-packages.tar.gz *.debExit from the Ubuntu machine (if SSH'd):
exitStep 2: Transfer the Packages from Ubuntu to Mac Using
scpTransfer the zipped package file to your Mac:
Replace
<ubuntu-user>with your Ubuntu username, and<ubuntu-ip>with your Ubuntu machine's IP address.Step 3: Transfer the Packages from Mac to the Target Ubuntu Machine Using
scpTransfer the zipped package file to the target Ubuntu machine:
Replace
<ubuntu-user>with your target Ubuntu username and<target-ubuntu-ip>with the IP address of the target Ubuntu machine.Step 4: Install the Docker Packages on the Target Ubuntu Machine
SSH into the target Ubuntu machine:
Unzip the transferred package file:
Navigate to the directory containing the
.debfiles:Install the Docker packages:
sudo dpkg -i *.debGJS2162 commentedon Aug 27, 2024
For now, i was having much delay with scp and rsync, so i used the below to get the tar.gz file :
GJS2162 commentedon Aug 27, 2024
Is there some fast way than rsync and scp to transfer file using ssh?
singhalkarun commentedon Aug 28, 2024
We can use any method that can utilse the ssh access that we have. Does gdown utilise ssh? As per my understanding it needs internet
Also, can you share total amount of data in GB (split modules wise - e.g., for docker, for gpu drivers, etc) we will need to transfer in case of offline deployment and what speed you are getting during scp. We can accordingly plan on timelines in future for offline deployments.
singhalkarun commentedon Aug 28, 2024
@GJS2162 to share
A list of components (system packages, docker images, etc)
Summarize process for 1 package installation and 1 docker image (raise a pr to add docs/no-internet-deployment.md)
@singhalkarun to prepare a list of domain names that are required during runtime (not a part of this ticket)
singhalkarun commentedon Aug 29, 2024