blocks|key|1848646|text|想法:也许你可以用预先计算好的值制作一个(大的)查找表？|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1848647|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Idea: Perhaps you can make a (large) lookup table with the values pre-calculated?

blocks|key|2062724|text|在1亿次调用时，我会开始怀疑分析器的开销是否会影响结果。将计算结果替换为no-op，看看是否仍报告消耗了60%25的执行时间……|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2062725|或者更好的是，创建一些测试数据，并使用秒表计时器来分析大约一百万个呼叫。|2062726|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

At 100 million calls, i'd start to wonder if profiler overhead isn't skewing your results. Replace the calculation with a no-op and see if it is still reported to consume 60% of the execution time... 

Or better yet, create some test data and use a stopwatch timer to profile a million or so calls.

blocks|key|2062738|text|在谷歌上搜索，我找到了Sigmoid函数的另一个实现。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2062739|public+double+Sigmoid(double+x)
{
+++return+2+/+(1+%2B+Math.Exp(-2+*+x))+-+1;
}|code-block|syntax|javascript|2062740|这是否符合您的需求？是不是更快？|2062741|http://dynamicnotions.blogspot.com/2008/09/sigmoid-function-in-c.html|offset|length|2062742|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|0|1X|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|U|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|V|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|W|8|@]|9|@$K|X|L|Y|1|Z]]|A|$]]|$1|M|3|-4|5|6|7|10|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|J]]]]

Doing a Google search, I found an alternative implementation of the Sigmoid function.

<pre><code>public double Sigmoid(double x)
{
 return 2 / (1 + Math.Exp(-2 * x)) - 1;
}
</code></pre>

Is that correct for your needs? Is it faster?

<a href="http://dynamicnotions.blogspot.com/2008/09/sigmoid-function-in-c.html" rel="nofollow noreferrer">http://dynamicnotions.blogspot.com/2008/09/sigmoid-function-in-c.html</a>

blocks|key|1848755|text|女高音有一些很好的优化你的电话：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1848756|public+static+float+Sigmoid(double+value)+
{
++++float+k+=+Math.Exp(value);
++++return+k+/+(1.0f+%2B+k);
}|code-block|syntax|javascript|1848757|如果您尝试查找表，并且发现它使用了太多的内存，那么您可以始终查看每个连续调用的参数值，并使用一些缓存技术。|1848758|例如，尝试缓存最后一个值和结果。如果下一次调用与上一次调用具有相同的值，则不需要计算它，因为您已经缓存了上一次结果。如果当前调用与上一次调用相同，即使100次中只有1次，您可能会为自己节省100万次计算。|1848759|或者，您可能会发现，在10个连续调用中，value参数平均相同2次，因此您可以尝试缓存最后10个值/答案。|1848760|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|R|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Soprano had some nice optimizations your call:

<pre><code>public static float Sigmoid(double value) 
{
 float k = Math.Exp(value);
 return k / (1.0f + k);
}
</code></pre>

If you try a lookup table and find it uses too much memory you could always looking at the value of your parameter for each successive calls and employing some caching technique. 

For example try caching the last value and result. If the next call has the same value as the previous one, you don't need to calculate it as you'd have cached the last result. If the current call was the same as the previous call even 1 out of a 100 times, you could potentially save yourself 1 million calculations.

Or, you may find that within 10 successive calls, the value parameter is on average the same 2 times, so you could then try caching the last 10 values/answers.

blocks|key|1848850|text|1)你只在一个地方调用吗？如果是这样，您可以通过将代码移出该函数并将其放在通常调用Sigmoid函数的位置来获得少量的性能。我不喜欢这个想法在代码可读性和组织方面，但是当你需要获得最后的性能提升时，这可能会有所帮助，因为我认为函数调用需要在堆栈上推入/弹出寄存器，如果你的代码都是内联的，就可以避免这种情况。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1848851|2)我不知道这是否有帮助，但尝试将函数参数设置为ref参数。看看它是不是更快。我会建议让它成为const+(如果这是在c%2B%2B中，这将是一个优化)，但是c#不支持const参数。|1848852|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

1) Do you call this from only one place? If so, you may gain a small amount of performance by moving the code out of that function and just putting it right where you would normally have called the Sigmoid function. I don't like this idea in terms of code readability and organization but when you need to get every last performance gain, this might help because I think function calls require a push/pop of registers on the stack, which could be avoided if your code was all inline.

2) I have no idea if this might help but try making your function parameter a ref parameter. See if it's faster. I would have suggested making it const (which would have been an optimization if this were in c++) but c# doesn't support const parameters.

blocks|key|2062847|text|您还可以考虑尝试使用评估成本更低的替代激活函数。例如：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2062848|f(x)+=+(3x+-+x**3)/2|code-block|syntax|javascript|2062849|(可以将其分解为|2062850|f(x)+=+x*(3+-+x*x)/2|2062851|减少一次乘法)。这个函数具有奇数对称性，并且它的导数是平凡的。将其用于神经网络需要通过除以输入总数来归一化输入总和(将域限制为-1..1，这也是范围)。|2062852|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

You might also consider experimenting with alternative activation functions which are cheaper to evaluate. For example:

<pre><code>f(x) = (3x - x**3)/2
</code></pre>

(which could be factored as

<pre><code>f(x) = x*(3 - x*x)/2
</code></pre>

for one less multiplication). This function has odd symmetry, and its derivative is trivial. Using it for a neural network requires normalizing the sum-of-inputs by dividing by the total number of inputs (limiting the domain to [-1..1], which is also range).

blocks|key|2062922|text|如果您能够与C%2B%2B进行互操作，则可以考虑将所有值存储在一个数组中，并使用SSE遍历它们，如下所示：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2062923|void+sigmoid_sse(float+*a_Values,+float+*a_Output,+size_t+a_Size){
++++__m128*+l_Output+=+(__m128*)a_Output;
++++__m128*+l_Start++=+(__m128*)a_Values;
++++__m128*+l_End++++=+(__m128*)(a_Values+%2B+a_Size);

++++const+__m128+l_One++++++++=+_mm_set_ps1(1.f);
++++const+__m128+l_Half+++++++=+_mm_set_ps1(1.f+/+2.f);
++++const+__m128+l_OneOver6+++=+_mm_set_ps1(1.f+/+6.f);
++++const+__m128+l_OneOver24++=+_mm_set_ps1(1.f+/+24.f);
++++const+__m128+l_OneOver120+=+_mm_set_ps1(1.f+/+120.f);
++++const+__m128+l_OneOver720+=+_mm_set_ps1(1.f+/+720.f);
++++const+__m128+l_MinOne+++++=+_mm_set_ps1(-1.f);

++++for(__m128+*i+=+l_Start;+i+<+l_End;+i%2B%2B){
++++++++//+1.0+/+(1.0+%2B+Math.Pow(Math.E,+-value))
++++++++//+1.0+/+(1.0+%2B+Math.Exp(-value))

++++++++//+value+=+*i+so+we+need+-value
++++++++__m128+value+=+_mm_mul_ps(l_MinOne,+*i);

++++++++//+exp+expressed+as+inifite+series+1+%2B+x+%2B+(x+%5E+2+/+2!)+%2B+(x+%5E+3+/+3!)+...
++++++++__m128+x+=+value;

++++++++//+result+in+l_Exp
++++++++__m128+l_Exp+=+l_One;+//+=+1

++++++++l_Exp+=+_mm_add_ps(l_Exp,+x);+//+%2B=+x

++++++++x+=+_mm_mul_ps(x,+x);+//+=+x+%5E+2
++++++++l_Exp+=+_mm_add_ps(l_Exp,+_mm_mul_ps(l_Half,+x));+//+%2B=+(x+%5E+2+*+(1+/+2))

++++++++x+=+_mm_mul_ps(value,+x);+//+=+x+%5E+3
++++++++l_Exp+=+_mm_add_ps(l_Exp,+_mm_mul_ps(l_OneOver6,+x));+//+%2B=+(x+%5E+3+*+(1+/+6))

++++++++x+=+_mm_mul_ps(value,+x);+//+=+x+%5E+4
++++++++l_Exp+=+_mm_add_ps(l_Exp,+_mm_mul_ps(l_OneOver24,+x));+//+%2B=+(x+%5E+4+*+(1+/+24))

#ifdef+MORE_ACCURATE

++++++++x+=+_mm_mul_ps(value,+x);+//+=+x+%5E+5
++++++++l_Exp+=+_mm_add_ps(l_Exp,+_mm_mul_ps(l_OneOver120,+x));+//+%2B=+(x+%5E+5+*+(1+/+120))

++++++++x+=+_mm_mul_ps(value,+x);+//+=+x+%5E+6
++++++++l_Exp+=+_mm_add_ps(l_Exp,+_mm_mul_ps(l_OneOver720,+x));+//+%2B=+(x+%5E+6+*+(1+/+720))

#endif

++++++++//+we've+calculated+exp+of+-i
++++++++//+now+we+only+need+to+do+the+'1.0+/+(1.0+%2B+...'+part
++++++++*l_Output%2B%2B+=+_mm_rcp_ps(_mm_add_ps(l_One,++l_Exp));
++++}
}|code-block|syntax|javascript|2062924|但是，请记住，您将使用的数组应该使用_aligned_malloc(some_size+*+sizeof(float)，16)进行分配，因为SSE需要与边界对齐的内存。|2062925|使用SSE，我可以在大约半秒内计算出所有1亿个元素的结果。然而，一次分配这么多的内存将花费您近三分之二的at，所以我建议一次处理更多但更小的数组。您甚至可能想要考虑使用具有100K或更多元素的双缓冲方法。|2062926|此外，如果元素的数量开始大幅增加，您可能希望选择在图形处理器上处理这些东西(只需创建一维float4纹理并运行非常简单的片段着色器)。|2062927|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|R|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

If you're able to interop with C++, you could consider storing all the values in an array and loop over them using SSE like this:

<pre><code>void sigmoid_sse(float *a_Values, float *a_Output, size_t a_Size){
 __m128* l_Output = (__m128*)a_Output;
 __m128* l_Start = (__m128*)a_Values;
 __m128* l_End = (__m128*)(a_Values + a_Size);

 const __m128 l_One = _mm_set_ps1(1.f);
 const __m128 l_Half = _mm_set_ps1(1.f / 2.f);
 const __m128 l_OneOver6 = _mm_set_ps1(1.f / 6.f);
 const __m128 l_OneOver24 = _mm_set_ps1(1.f / 24.f);
 const __m128 l_OneOver120 = _mm_set_ps1(1.f / 120.f);
 const __m128 l_OneOver720 = _mm_set_ps1(1.f / 720.f);
 const __m128 l_MinOne = _mm_set_ps1(-1.f);

 for(__m128 *i = l_Start; i &lt; l_End; i++){
 // 1.0 / (1.0 + Math.Pow(Math.E, -value))
 // 1.0 / (1.0 + Math.Exp(-value))

 // value = *i so we need -value
 __m128 value = _mm_mul_ps(l_MinOne, *i);

 // exp expressed as inifite series 1 + x + (x ^ 2 / 2!) + (x ^ 3 / 3!) ...
 __m128 x = value;

 // result in l_Exp
 __m128 l_Exp = l_One; // = 1

 l_Exp = _mm_add_ps(l_Exp, x); // += x

 x = _mm_mul_ps(x, x); // = x ^ 2
 l_Exp = _mm_add_ps(l_Exp, _mm_mul_ps(l_Half, x)); // += (x ^ 2 * (1 / 2))

 x = _mm_mul_ps(value, x); // = x ^ 3
 l_Exp = _mm_add_ps(l_Exp, _mm_mul_ps(l_OneOver6, x)); // += (x ^ 3 * (1 / 6))

 x = _mm_mul_ps(value, x); // = x ^ 4
 l_Exp = _mm_add_ps(l_Exp, _mm_mul_ps(l_OneOver24, x)); // += (x ^ 4 * (1 / 24))

#ifdef MORE_ACCURATE

 x = _mm_mul_ps(value, x); // = x ^ 5
 l_Exp = _mm_add_ps(l_Exp, _mm_mul_ps(l_OneOver120, x)); // += (x ^ 5 * (1 / 120))

 x = _mm_mul_ps(value, x); // = x ^ 6
 l_Exp = _mm_add_ps(l_Exp, _mm_mul_ps(l_OneOver720, x)); // += (x ^ 6 * (1 / 720))

#endif

 // we've calculated exp of -i
 // now we only need to do the '1.0 / (1.0 + ...' part
 *l_Output++ = _mm_rcp_ps(_mm_add_ps(l_One, l_Exp));
 }
}
</code></pre>

However, remember that the arrays you'll be using should be allocated using _aligned_malloc(some_size * sizeof(float), 16) because SSE requires memory aligned to a boundary.

Using SSE, I can calculate the result for all 100 million elements in around half a second. However, allocating that much memory at a time will cost you nearly two-third of a gigabyte so I'd suggest processing more but smaller arrays at a time. You might even want to consider using a double buffering approach with 100K elements or more.

Also, if the number of elements starts to grow considerably you might want to choose to process these things on the GPU (just create a 1D float4 texture and run a very trivial fragment shader).

blocks|key|2062956|text|在我的脑海中，this+paper+explains+a+way+of+approximating+the+exponential+by+abusing+floating+point+(点击右上角的链接)，但是我不知道它在.NET中是否对你有很大的用处。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2062957|此外，还有一点:为了快速训练大型网络，您使用的逻辑sigmoid非常糟糕。请参阅Efficient+Backprop+by+LeCun+et+al的4.4节，并使用以零为中心的内容(实际上，请阅读整篇文章，它非常有用)。|2062958|entityMap|0|LINK|mutability|MUTABLE|url|http://citeseerx.ist.psu.edu/icons/pdf.gif;jsessionid=EFB25A837B2026808E5BEF018B36521E|1|http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf^0|7|2C|0|0|14|X|1|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]]|C|$]]|$1|D|3|E|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]]|C|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]|N|$5|I|J|K|C|$L|O]]]]

Off the top of my head, <a href="http://citeseerx.ist.psu.edu/icons/pdf.gif;jsessionid=EFB25A837B2026808E5BEF018B36521E" rel="nofollow noreferrer">this paper explains a way of approximating the exponential by abusing floating point</a>, (click the link in the top right for PDF) but I don't know if it'll be of much use to you in .NET.

Also, another point: for the purpose of training large networks quickly, the logistic sigmoid you're using is pretty terrible. See section 4.4 of <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf" rel="nofollow noreferrer">Efficient Backprop by LeCun et al</a> and use something zero-centered (actually, read that whole paper, it's immensely useful).

blocks|key|1848987|text|如果你需要一个巨大的速度提升，你可能会考虑使用(ge)force并行化函数。我的意思是，使用DirectX来控制显卡来为你做这件事。我不知道怎么做，但我见过人们使用图形卡进行各种计算。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1848988|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

If you need a giant speed boost, you could probably look into parallelizing the function using the (ge)force. IOW, use DirectX to control the graphics card into doing it for you. I have no idea how to do this, but I've seen people use graphics cards for all kinds of calculations.

blocks|key|2063012|text|这有点离题，但出于好奇，我用Java语言实现了与C、C#和F#相同的实现。我就把这个留在这里，以防其他人好奇。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2063013|结果：|2063014|$+javac+LUTTest.java+&&+java+LUTTest
Max+deviation+is+0.001664
10%5E7+iterations+using+sigmoid1()+took+1398+ms
10%5E7+iterations+using+sigmoid2()+took+177+ms|code-block|syntax|javascript|2063015|我认为在我的例子中对C#的改进是由于Java比Mono在OS+X上进行了更好的优化。在类似的MS.NET实现上(如果有人想要发布比较数字，与Java6相比)，我想结果会有所不同。|2063016|代码：|2063017|public+class+LUTTest+{
++++private+static+final+float+SCALE+=+320.0f;
++++private+static+final++int+RESOLUTION+=+2047;
++++private+static+final++float+MIN+=+-RESOLUTION+/+SCALE;
++++private+static+final++float+MAX+=+RESOLUTION+/+SCALE;

++++private+static+final+float[]+lut+=+initLUT();

++++private+static+float[]+initLUT()+{
++++++++float[]+lut+=+new+float[RESOLUTION+%2B+1];

++++++++for+(int+i+=+0;+i+<+RESOLUTION+%2B+1;+i%2B%2B)+{
++++++++++++lut[i]+=+(float)(1.0+/+(1.0+%2B+Math.exp(-i+/+SCALE)));
++++++++}
++++++++return+lut;
++++}

++++public+static+float+sigmoid1(double+value)+{
++++++++return+(float)+(1.0+/+(1.0+%2B+Math.exp(-value)));
++++}

++++public+static+float+sigmoid2(float+value)+{
++++++++if+(value+<=+MIN)+return+0.0f;
++++++++if+(value+>=+MAX)+return+1.0f;
++++++++if+(value+>=+0)+return+lut[(int)(value+*+SCALE+%2B+0.5f)];
++++++++return+1.0f+-+lut[(int)(-value+*+SCALE+%2B+0.5f)];
++++}

++++public+static+float+error(float+v0,+float+v1)+{
++++++++return+Math.abs(v1+-+v0);
++++}

++++public+static+float+testError()+{
++++++++float+emax+=+0.0f;
++++++++for+(float+x+=+-10.0f;+x+<+10.0f;+x%2B=+0.00001f)+{
++++++++++++float+v0+=+sigmoid1(x);
++++++++++++float+v1+=+sigmoid2(x);
++++++++++++float+e+=+error(v0,+v1);
++++++++++++if+(e+>+emax)+emax+=+e;
++++++++}
++++++++return+emax;
++++}

++++public+static+long+sigmoid1Perf()+{
++++++++float+y+=+0.0f;
++++++++long+t0+=+System.currentTimeMillis();
++++++++for+(int+i+=+0;+i+<+10;+i%2B%2B)+{
++++++++++++for+(float+x+=+-5.0f;+x+<+5.0f;+x%2B=+0.00001f)+{
++++++++++++++++y+=+sigmoid1(x);
++++++++++++}
++++++++}
++++++++long+t1+=+System.currentTimeMillis();
++++++++System.out.printf("",y);
++++++++return+t1+-+t0;
++++}++++

++++public+static+long+sigmoid2Perf()+{
++++++++float+y+=+0.0f;
++++++++long+t0+=+System.currentTimeMillis();
++++++++for+(int+i+=+0;+i+<+10;+i%2B%2B)+{
++++++++++++for+(float+x+=+-5.0f;+x+<+5.0f;+x%2B=+0.00001f)+{
++++++++++++++++y+=+sigmoid2(x);
++++++++++++}
++++++++}
++++++++long+t1+=+System.currentTimeMillis();
++++++++System.out.printf("",y);
++++++++return+t1+-+t0;
++++}++++

++++public+static+void+main(String[]+args)+{

++++++++System.out.printf("Max+deviation+is+%25f\n",+testError());
++++++++System.out.printf("10%5E7+iterations+using+sigmoid1()+took+%25d+ms\n",+sigmoid1Perf());
++++++++System.out.printf("10%5E7+iterations+using+sigmoid2()+took+%25d+ms\n",+sigmoid2Perf());
++++}
}|2063018|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/412019/code-optimizations#412176|1|https://stackoverflow.com/questions/412019/code-optimizations#414152|2|https://stackoverflow.com/questions/412019/code-optimizations#416180^0|O|1|0|Q|2|1|T|2|2|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@$A|13|B|14|1|15]|$A|16|B|17|1|18]|$A|19|B|1A|1|1B]]|C|$]]|$1|D|3|E|5|6|7|1C|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|1D|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|1E|8|@]|9|@]|C|$]]|$1|M|3|N|5|6|7|1F|8|@]|9|@]|C|$]]|$1|O|3|P|5|H|7|1G|8|@]|9|@]|C|$I|J]]|$1|Q|3|-4|5|6|7|1H|8|@]|9|@]|C|$]]]|R|$S|$5|T|U|V|C|$W|X]]|Y|$5|T|U|V|C|$W|Z]]|10|$5|T|U|V|C|$W|11]]]]

This is slightly off topic, but just out of curiosity, I did the same implementation as the one in <a href="https://stackoverflow.com/questions/412019/code-optimizations#412176">C</a>, <a href="https://stackoverflow.com/questions/412019/code-optimizations#414152">C#</a> and <a href="https://stackoverflow.com/questions/412019/code-optimizations#416180">F#</a> in Java. I'll just leave this here in case someone else is curious.

Result:

<pre><code>$ javac LUTTest.java &amp;&amp; java LUTTest
Max deviation is 0.001664
10^7 iterations using sigmoid1() took 1398 ms
10^7 iterations using sigmoid2() took 177 ms
</code></pre>

I suppose the improvement over C# in my case is due to Java being better optimized than Mono for OS X. On a similar MS .NET-implementation (vs. Java 6 if someone wants to post comparative numbers) I suppose the results would be different.

Code:

<pre><code>public class LUTTest {
 private static final float SCALE = 320.0f;
 private static final int RESOLUTION = 2047;
 private static final float MIN = -RESOLUTION / SCALE;
 private static final float MAX = RESOLUTION / SCALE;

 private static final float[] lut = initLUT();

 private static float[] initLUT() {
 float[] lut = new float[RESOLUTION + 1];

 for (int i = 0; i &lt; RESOLUTION + 1; i++) {
 lut[i] = (float)(1.0 / (1.0 + Math.exp(-i / SCALE)));
 }
 return lut;
 }

 public static float sigmoid1(double value) {
 return (float) (1.0 / (1.0 + Math.exp(-value)));
 }

 public static float sigmoid2(float value) {
 if (value &lt;= MIN) return 0.0f;
 if (value &gt;= MAX) return 1.0f;
 if (value &gt;= 0) return lut[(int)(value * SCALE + 0.5f)];
 return 1.0f - lut[(int)(-value * SCALE + 0.5f)];
 }

 public static float error(float v0, float v1) {
 return Math.abs(v1 - v0);
 }

 public static float testError() {
 float emax = 0.0f;
 for (float x = -10.0f; x &lt; 10.0f; x+= 0.00001f) {
 float v0 = sigmoid1(x);
 float v1 = sigmoid2(x);
 float e = error(v0, v1);
 if (e &gt; emax) emax = e;
 }
 return emax;
 }

 public static long sigmoid1Perf() {
 float y = 0.0f;
 long t0 = System.currentTimeMillis();
 for (int i = 0; i &lt; 10; i++) {
 for (float x = -5.0f; x &lt; 5.0f; x+= 0.00001f) {
 y = sigmoid1(x);
 }
 }
 long t1 = System.currentTimeMillis();
 System.out.printf("",y);
 return t1 - t0;
 } 

 public static long sigmoid2Perf() {
 float y = 0.0f;
 long t0 = System.currentTimeMillis();
 for (int i = 0; i &lt; 10; i++) {
 for (float x = -5.0f; x &lt; 5.0f; x+= 0.00001f) {
 y = sigmoid2(x);
 }
 }
 long t1 = System.currentTimeMillis();
 System.out.printf("",y);
 return t1 - t0;
 } 

 public static void main(String[] args) {

 System.out.printf("Max deviation is %f\n", testError());
 System.out.printf("10^7 iterations using sigmoid1() took %d ms\n", sigmoid1Perf());
 System.out.printf("10^7 iterations using sigmoid2() took %d ms\n", sigmoid2Perf());
 }
}
</code></pre>

blocks|key|2063046|text|有一个更快的函数可以做非常相似的事情：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2063047|x+/+(1+%2B+abs(x))+-+TAHN的快速替代品|offset|length|style|CODE|2063048|类似地：|2063049|x+/+(2+%2B+2+*+abs(x))+%2B+0.5+-+SIGMOID的快速替代|2063050|Compare+plots+with+actual+sigmoid|2063051|entityMap|0|LINK|mutability|MUTABLE|url|http://fooplot.com/#W3sidHlwZSI6MCwiZXEiOiJ4LygyKzJhYnMoeCkpKzAuNSIsImNvbG9yIjoiIzAwMDAwMCJ9LHsidHlwZSI6MCwiZXEiOiIxLygxK01hdGguRV4oLXgpKSIsImNvbG9yIjoiIzAwMDAwMCJ9LHsidHlwZSI6MTAwMH1d^0|0|0|G|0|0|0|Q|0|0|X|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|W|8|@$D|X|E|Y|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|Z|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|10|8|@$D|11|E|12|F|G]]|9|@]|A|$]]|$1|L|3|M|5|6|7|13|8|@]|9|@$D|14|E|15|1|16]]|A|$]]|$1|N|3|-4|5|6|7|17|8|@]|9|@]|A|$]]]|O|$P|$5|Q|R|S|A|$T|U]]]]

There are a much faster functions that do very similar things:

<code>x / (1 + abs(x))</code> – fast replacement for TAHN

And similarly:

<code>x / (2 + 2 * abs(x)) + 0.5</code> - fast replacement for SIGMOID

<a href="http://fooplot.com/#W3sidHlwZSI6MCwiZXEiOiJ4LygyKzJhYnMoeCkpKzAuNSIsImNvbG9yIjoiIzAwMDAwMCJ9LHsidHlwZSI6MCwiZXEiOiIxLygxK01hdGguRV4oLXgpKSIsImNvbG9yIjoiIzAwMDAwMCJ9LHsidHlwZSI6MTAwMH1d" rel="nofollow">Compare plots with actual sigmoid</a>

blocks|key|2063068|text|我看到很多人都在尝试使用近似来使Sigmoid更快。但是，重要的是要知道，Sigmoid也可以使用tanh来表达，而不仅仅是exp。用这种方法计算Sigmoid比使用指数快大约5倍，并且通过使用这种方法，你不会近似任何东西，因此Sigmoid的原始行为保持不变。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2063069|++++public+static+double+Sigmoid(double+value)
++++{
++++++++return+0.5d+%2B+0.5d+*+Math.Tanh(value/2);
++++}|code-block|syntax|javascript|2063070|当然，并行化将是提高性能的下一步，但就原始计算而言，使用Math.Tanh比使用Math.Exp更快。|2063071|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

I've seen that a lot of people around here are trying to use approximation to make Sigmoid faster. However, it is important to know that Sigmoid can also be expressed using tanh, not only exp.
Calculating Sigmoid this way is around 5 times faster than with exponential, and by using this method you are not approximating anything, thus the original behaviour of Sigmoid is kept as-is.

<pre><code> public static double Sigmoid(double value)
 {
 return 0.5d + 0.5d * Math.Tanh(value/2);
 }
</code></pre>

Of course, parellization would be the next step to performance improvement, but as far as the raw calculation is concerned, using Math.Tanh is faster than Math.Exp.

blocks|key|2063171|text|记住，Sigmoid约束结果的范围在0到1之间。小于-10的值返回的值非常非常接近0.0。大于10的值返回的值非常非常接近1。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2063172|回到过去，当计算机不能很好地处理算术溢出/下溢时，使用if条件来限制计算是很常见的。如果我真的担心它的性能(或者基本上是Math的性能)，我会把你的代码改成老式的方式(注意限制)，这样它就不会不必要地调用Math：|2063173|public+double+Sigmoid(double+value)
{
++++if+(value+<+-45.0)+return+0.0;
++++if+(value+>+45.0)+return+1.0;
++++return+1.0+/+(1.0+%2B+Math.Exp(-value));
}|code-block|syntax|javascript|2063174|我意识到任何阅读此答案的人都可能参与了某种NN开发。请注意上述内容如何影响您的培训分数的其他方面。|2063175|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|O|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Remember, Sigmoid constraints results to range between 0 and 1. Values of smaller than about -10 return a value very, very close to 0.0. Values of greater than about 10 return a value very, very close to 1.

Back in the old days when computers couldn't handle arithmetic overflow/underflow that well, putting if conditions to limit the calculation was usual. If I were really concerned about its performance (or basically Math's performance), I would change your code to the old fashioned way (and mind the limits) so that it does not call Math unnecessarily:

<pre><code>public double Sigmoid(double value)
{
 if (value &lt; -45.0) return 0.0;
 if (value &gt; 45.0) return 1.0;
 return 1.0 / (1.0 + Math.Exp(-value));
}
</code></pre>

I realize anyone reading this answer may be involved in some sort of NN development. Be mindful of how the above affects the other aspects of your training scores.

I've been profiling an application all day long and, having optimized a couple bits of code, I'm left with this on my todo list. It's the activation function for a neural network, which gets called over a 100 million times. According to dotTrace, it amounts to about 60% of the overall function time.

How would you optimize this?

<pre><code>public static float Sigmoid(double value) {
 return (float) (1.0 / (1.0 + Math.Pow(Math.E, -value)));
}
</code></pre>

Math optimization in C#

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我一整天都在分析一个应用程序，在优化了几个代码后，我的待办事项清单上就剩下这个了。这是一个神经网络的激活函数，它被调用了超过1亿次。根据dotTrace的说法，它相当于整个功能时间的60%左右。您将如何优化此功能？public static float Sigmoid(double value) {    return...

问C#中的数学优化
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问C#中的数学优化EN