AWS EC2 (AMI)에 Java, Scala, Spark, Kafka, MongoDB, Redis, Node.js, Maven 설치

1. AWS EC2 접속

2. AMI 버전 확인

$ grep . /etc/*-release

/etc/os-release:NAME="Amazon Linux AMI"

/etc/os-release:VERSION="2017.03"

/etc/os-release:ID="amzn"

/etc/os-release:ID_LIKE="rhel fedora"

/etc/os-release:VERSION_ID="2017.03"

/etc/os-release:PRETTY_NAME="Amazon Linux AMI 2017.03"

/etc/os-release:ANSI_COLOR="0;33"

/etc/os-release:CPE_NAME="cpe:/o:amazon:linux:2017.03:ga"

/etc/os-release:HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

/etc/system-release:Amazon Linux AMI release 2017.03

3. Java 1.8 설치

$ java -version

java version "1.7.0_151"

OpenJDK Runtime Environment (amzn-2.6.11.0.74.amzn1-x86_64 u151-b00)

OpenJDK 64-Bit Server VM (build 24.151-b00, mixed mode)

$ sudo yum install java-1.8.0

$ sudo yum remove java-1.7.0

$ java -version

openjdk version "1.8.0_141"

OpenJDK Runtime Environment (build 1.8.0_141-b16)

OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

4. Scala 설치

$ wget https://downloads.lightbend.com/scala/2.12.3/scala-2.12.3.tgz

$ tar xzvf scala-2.12.3.tgz

$ sudo su -

# cd /home/ec2-user/

# mv scala-2.12.3 /usr/local/scala

# exit

$ sudo vi /etc/profile

export PATH=$PATH:/usr/local/scala/bin

$ source /etc/profile

$ scala -version

Scala code runner version 2.12.3 -- Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.

5. Spark 설치

* 주의 사항 : AWS t2.large 인스턴스 정도가 되어야 정상 동작함 (t2.small에서는 Spark Streaming 정상 동작 하지 않음)

$ wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

$ tar xvzf spark-2.2.0-bin-hadoop2.7.tgz

$ sudo su -

# cd /home/ec2-user/

# mv spark-2.2.0-bin-hadoop2.7 /usr/local/spark

# exit

$ sudo vi /etc/profile

export PATH=$PATH:/usr/local/spark/bin

$ source /etc/profile

$ spark-shell

6. Kafka 설치 및 데몬 실행

$ wget http://apache.mirror.cdnetworks.com/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz

$ tar xzvf kafka_2.11-0.11.0.0.tgz

$ ln -s kafka_2.11-0.11.0.0 kafka

$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

$ bin/kafka-server-start.sh -daemon config/server.properties

7. MongoDB 설치 및 서비스 실행

$ sudo su

# vi /etc/yum.repos.d/mongodb-org-3.4.repo

[mongodb-org-3.4]

name=MongoDB Repository

baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.4/x86_64/

gpgcheck=1

enabled=1

gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc

# yum install -y mongodb-org

# service mongod start

# mongo

# exit

8. Redis 설치 및 서비스 실행

$ sudo yum -y update

$ sudo yum -y install gcc make

$ sudo wget http://download.redis.io/redis-stable.tar.gz

$ tar xvzf redis-stable.tar.gz

$ cd redis-stable

$ sudo make install

$ sudo mkdir -p /etc/redis /var/lib/redis /var/redis/6379

$ sudo cp redis.conf /etc/redis/6379.conf

$ sudo vi /etc/redis/6379.conf

daemonize yes

logfile /var/log/redis_6379.log

dir /var/redis/6379

$ sudo wget https://raw.githubusercontent.com/saxenap/install-redis-amazon-linux-centos/master/redis-server

$ sudo mv redis-server /etc/init.d

$ sudo chmod 755 /etc/init.d/redis-server

$ sudo vi /etc/init.d/redis-server

REDIS_CONF_FILE="/etc/redis/6379.conf"

$ sudo chkconfig --add redis-server

$ sudo chkconfig --level 345 redis-server on

$ sudo service redis-server start

9. Node.js 개발 환경 구축

$ sudo yum install git-core

$ sudo yum install nodejs npm --enablerepo=epel

$ npm -v

1.3.6

$ sudo npm update -g npm

$ npm -v

5.3.0

$ npm install

$ sudo npm cache clean -f

$ sudo npm install -g n

$ sudo n 6.11.1

$ node -v

v0.10.48

재접속

$ node -v

v6.11.1

10. Maven 개발 환경 구축

$ sudo yum install java-1.8.0-openjdk-devel.x86_64

$ sudo alternatives --config java

$ sudo alternatives --config javac

$ javac -version

javac 1.8.0_141

$ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

$ sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo

$ sudo yum install -y apache-maven

$ mvn -v

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)

Maven home: /usr/share/apache-maven

Java version: 1.7.0_151, vendor: Oracle Corporation

Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151.x86_64/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "4.9.38-16.35.amzn1.x86_64", arch: "amd64", family: "unix"

------------------

<참고자료>

리눅스 버전 확인

https://zetawiki.com/wiki/%EB%A6%AC%EB%88%85%EC%8A%A4_%EC%A2%85%EB%A5%98_%ED%99%95%EC%9D%B8,_%EB%A6%AC%EB%88%85%EC%8A%A4_%EB%B2%84%EC%A0%84_%ED%99%95%EC%9D%B8

AMI에 JDK 1.8 설치

http://blog.naver.com/PostView.nhn?blogId=typez&logNo=221020775376&redirect=Dlog&widgetTypeCall=true

스칼라 및 스파크 설치

http://www.w3ii.com/ko/apache_spark/apache_spark_installation.html

카프카 설치

http://www.popit.kr/kafka-%EC%9A%B4%EC%98%81%EC%9E%90%EA%B0%80-%EB%A7%90%ED%95%98%EB%8A%94-%EC%B2%98%EC%9D%8C-%EC%A0%91%ED%95%98%EB%8A%94-kafka/

https://blog.knoldus.com/2017/04/19/installing-and-running-kafka-on-aws-instance-centos/

몽고디비 설치

http://chichi.space/2017/05/12/%ED%95%9C%EB%B2%88%EC%97%90-%EB%81%9D%EB%82%B4%EB%8A%94-AWS-EC2%EC%97%90-MongoDB-%EC%84%A4%EC%B9%98%ED%95%98%EA%B3%A0-%EB%B3%B4%EC%95%88%EC%84%A4%EC%A0%95%ED%95%98%EA%B8%B0/

레디스 설치

http://mygumi.tistory.com/133

웹 프론트 개발 환경 구축

http://tbang.tistory.com/123

Node.js 최신 버전 설치

http://goosia.com/?p=85

NPM 최신 버전 설치

https://askubuntu.com/questions/562417/how-do-you-update-npm-to-the-latest-version

$ npm -v
2.15.1
$ sudo npm update -g npm
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
npm@3.10.9 /usr/local/lib/node_modules/npm
$ npm -v
3.10.9

Maven 설치

http://bhargavamin.com/how-to-do/install-jenkins-on-amazon-linux-aws/

sudo yum install -y git  java-1.8.0-openjdk-devel aws-cli

sudo alternatives --config java

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo

sudo yum install -y apache-maven

mvn –v

* AWS EC2 t2.micro Swap 할당하는 방법

https://m.blog.naver.com/PostView.nhn?blogId=sory1008&logNo=220808623133&proxyReferer=https%3A%2F%2Fwww.google.co.kr%2F

$ dd if=/dev/zero of=/swapfile bs=1M count=1024 $ mkswap /swapfile $ swapon /swapfile $ echo "/swapfile swap swap defaults 0 0" >> /etc/fstab

http://wiki.pchero21.com/wiki/Swap_memory

Swap file

이미 파일 시스템을 구성한 이후라면, 스왑 파티션을 생성하는 것이 부담스러울 수 있다. 이런 경우, 간단하게 스왑 파일을 생성하여 스왑 영역으로 사용하면 된다.

pchero@earth:/home$ sudo dd if=/dev/zero of=/home/swapfile bs=1024 count=2000000
2000000+0 records in
2000000+0 records out
2048000000 bytes (2.0 GB) copied, 6.53252 s, 314 MB/s
 
pchero@earth:/home$ ls -sh
total 2.0G
4.0K pchero  2.0G swapfile
 
pchero@earth:/home$ sudo mkswap /home/swapfile 
Setting up swapspace version 1, size = 1999996 KiB
no label, UUID=7373fad5-bfca-4bb9-b8e7-2a92cf17fa7d
 
pchero@earth:/home$ sudo swapon /home/swapfile
swapon: /home/swapfile: insecure permissions 0644, 0600 suggested.
 
pchero@earth:/home$ free -m
             total       used       free     shared    buffers     cached
Mem:          1000        935         64         82          6        338
-/+ buffers/cache:        590        409
Swap:         1953          0       1953

파일 생성 후, /etc/fstab 에 다음과 같이 추가를 해주자.

# another swap file
/home/swapfile  none    swap    sw      0       0