抱歉,你查看的文章不存在

aws 上面最新最全的GPU深度学习镜像-避免手工安装各种各样的软件

We Have To Go Deeper

aws p2.xlarge GPU optimized deep learning cluster-grenade.

All installations are bleeding-edge as of Halloween 2016

Region-specific AMIs

ami-a195cfb6 - US East (N. Virginia)

ami-86277de3 - US East (Ohio)

ami-3e22685e - US West (N. California)

ami-da3096ba - US West (Oregon)

ami-afa3e9dc - EU (Ireland)

ami-8f906be0 - EU (Frankfurt)

ami-32d37e53 - Asia Pacific (Tokyo)

ami-6a20f404 - Asia Pacific (Seoul)

ami-20e54543 - Asia Pacific (Singapore)

ami-6f2b170c - Asia Pacific (Sydney)

ami-8da5d1e2 - Asia Pacific (Mumbai)

ami-b3a438df - South America (São Paulo)

Greetings, traveler.

Here we have a Deep Learning system image (AWS AMI) custom built for the p2.xlarge system, which at ~1$ per hour is a GREAT price for a machine as powerful as it is. I've had my share of difficulties with building various modules/packages/drivers personally, so I thought I would share this for any fellow ml/dl/ai researchers/professionals who feel they might benefit from not having to go through the lengthy process of debugging/hacking that I did..

Design Philosophy

I wanted to build a flexible system that would allow myself and others to seamlessly explore whatever machine learning / deep learning system I wanted to, while side-stepping the plethora of dependency conflict issues / custom build issues that inevitably seem to arise each time I try a new package. To that end, I wanted to start off with a clean system, and do a simul-build of all of the latest, bleeding-edge packages around. I'm sure I inevitably missed a few things here and there, but feel free to shoot me an email at mark.woods89 (at) gmail (dot) com if theres something you'd like to see added to this image.

This build is somewhat weighted towards python-facing systems because of my background, but I tried to also include many of the other deep learning environments that I've been hearing of around town.

Some hilights of the build:

CUDA 8.0

cuDNN 5.1

Microsoft CNTK V2

OpenCV 3.1

Spark 2.0

Caffe 1.0

Tensorflow

Theano

Keras

Torch 7

R

Scala

Julia

also it comes with a virtual desktop set up via vncserver in case you're into that kind of thing.

Additional Notes

All installed packages/modules/languages/etc have been tested and confirmed to work properly with GPU (at least for simple tests).

Also, the GPU has been max-clocked ala:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html#optimize_gpu

Getting Started

1) Get an aws account, initialize it with this machine image from 'Community AMIs' : (eg., ami-a195cfb6 for US East ( N. Virginia )

2) user credentials:

username: icarus

password: changetheworld

3) Reboot the instance. (this starts the init scripts properly)

4) run '$ aws configure' so that you can use the super handy-dandy aws-cli.

5) To access the vncserver-hosted desktop, I personally like VNC Viewer, since it's pretty lightweight and simple to use. You can find it here: https://www.realvnc.com/download/viewer/

(I still haven't quite worked out the issue with the super button not opening the search menu properly, but oh well. If you have tips on getting that to work, let me know.)

Then, to load the desktop, just boot up VNC Viewer, put your public DNS in the 'VNC Server:' box, followed by ':1' (without quotes) to access the virtual desktop loaded on screen :1.

password: godeeper

6) If you are a master sysadmin and you think the word 'desktop' is dirty, you can disable the auto-load script at /etc/init.d/vncserver . Alternatively, you can configure it as you like there. It's currently set to create a 1920x1080 desktop at 24-bit color depth.

7) A few things still dont work quite properly due to the system being a headless server with the bleeding-edge drivers required for the installed packages. For example, gnome-control-center will give an ominous error about a BadRequest from the X Window System. I honestly recommend not trying to get too deep into debugging that, it's a pain. Instead, there are packages around that will allow you to do what you need to in the system settings, or as an alternative:

Log into the instance via something like putty + xming, '$ vncserver -kill :1' to shut down the virtual desktop and run gnome-control-center from your terminal login. That will load a gui interface for that if you need it. (If anyone knows a way to get it working in the vnc desktop, I'd be happy to hear. For time's sake though, I chose to just adopt the workaround)

8) Do all the machine learning.

In general, a decent way to use this system is just to assume that what you need is already installed on it and configured to work.

Just try some of these commands in the terminal:

$ th

$ pyspark

$ digits-devserver

$ cntk

$ python

>>> import caffe

>>> import tensorflow

>>> import theano

>>> import keras

9) For more information on build paths and such, please reference ~/.bashrc . I tried to include most (if not all) of the relevant directories there.

10) Additionally, I set up a little script in the home directory to run commands sequentially followed by shutting down the instance. It's not pretty code, but for something that took less than 2min to make, it comes in handy when you want to get more work done in a night, but dont want to waste $$ by leaving your instance running. Don't forget to alter it so that it uses your own instance ID !

And that's it! Enjoy folks!

If you find this useful, please shoot me an email over at mark.woods89 (at) gmail (dot) com . Hopefully it will save some of y'all a lot of the headaches I went through trying to get it set up myself.

Oh yeah, and if you're just getting into machine learning, remember: change the world.

System

aws p2.xlarge

Ubuntu 14.04 LTS (Trusty Tahr)

NVIDIA Tesla K80

50 Gb storage

AWS cli

aws-cli 1.11.10

NVIDIA things.

NVIDIA CUDA 8.0

NVIDIA cuDNN 5.1

NVIDIA CUB 1.5.5

NVIDIA DIGITS 5.1-dev

NVIDIA CNMeM 1.0 (?)

Pythony things.

Anaconda 2

python 2.7

opencv 3.1.0

spark 2.0.1

caffe 1.0.0-rc3

keras 1.1.0

tensorflow 0.11.0rc0

tf-learn 0.2.2

theano 0.8.2

Torch.

torch 7

See file luarocks_installs_versions.txt in the home directory of the system for included packages

Other languages/packages for ml/dl

R 3.0.2

shiny 0.14.1

Scala 2.11.8

scala sbt 0.13.12

Julia 0.5

IntelliJ IDEA 2016.2.5 # For DeepLearning4J

maven 3.0.5

Microsoft CNTK

kaldi 1.0(?)

Just a few BLAS/math/etc libs...

ATLAS 3.10.1

OpenBLAS 0.2.20

MKL

Microsoft MKLCustom

openMPI 1.10.3

zlib 1.2.8

libzip 3.0

Boost 1.54.0

protobuf 3.1.0

原文发布于微信公众号 - CreateAMind(createamind)

原文发表时间:2016-11-01

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

编辑于

CreateAMind

544 篇文章32 人订阅

相关文章

来自专栏WOLFRAM

by 落霜枫舞

1543
来自专栏HansBug's Lab

1623: [Usaco2008 Open]Cow Cars 奶牛飞车

1623: [Usaco2008 Open]Cow Cars 奶牛飞车 Time Limit: 5 Sec  Memory Limit: 64 MB Submi...

2857
来自专栏linux驱动个人学习

高通msm8909耳机调试

1、DTS相应修改: DTS相关代码:kernel/arch/arm/boot/dts/qcom/msm8909-qrd-skuc.dtsi: 1 s...

1K5
来自专栏码匠的流水账

聊聊rocketmq的RequestTask

org/apache/rocketmq/remoting/netty/RequestTask.java

1862
来自专栏一个会写诗的程序员的博客

java.base.jmod

/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/jmods$ jmod list java....

1562
来自专栏腾讯数据中心

敬请收藏:数据中心常用标识的中英文对照

中国的数据中心在不断走向国际化,同时数据中心内的关键标识也逐渐采取了中英文双语标识。 今天,我们整理出腾讯数据中心内部使用的中英文标识对照。敬请收藏以备后续参考...

6304
来自专栏Hongten

My Notepad

I have spent near more two weeks to write this Notepad application. At this mome...

1112
来自专栏SAP最佳业务实践

SAP S/4 HANA新变化-MM物料管理

Co-Deployment of SAP SRM 集成SRM The functional scope of SAPSupplier Relationship...

3994
来自专栏叁金大数据

EmguCV学习——视频与图片互转

其实视频转图片在上篇文章中已经有些眉目了,其实就是按帧读取视频,然后把帧保存就ok。然后自己再加个进度条美化一下。。。这代码简单易懂,还是直接上代码吧。

2241
来自专栏从零开始的linux

linux基础命令6

tr命令 # tr 'a-z' 'A-Z' < /etc/fstab # # /ETC/FSTAB # CREATED BY ANACONDA ON THU ...

2877

扫码关注云+社区

领取腾讯云代金券