기울기하강,gradient_descent

Redirected from page "descent"
아주 대충,
grad f는 f의 direction of steepest ascent.
그래서 그 부호,sign를 반대로 한
-grad f는 f의 direction of steepest descent?
see also 방향도함수,directional_derivative 델,del,나블라,nabla

근사,approximation법과 비슷한 다음단계로 나아가는 반복적 방법(반복,iteration, iterative method)
최적화하고자 하는(최소점을 찾고자 하는) 함수가 $f$ 라면
$x_{i+1} = x_i - \alpha \frac{df}{dx}(x_i)$
그리고 다변수함수로 일반화(확장)하면
$x_{i+1} = x_i - \alpha \nabla f(x_i)$
(angeloyeo)
저기서 부호가 $-$ 대신 $+$ 로 되면 gradient ascent.
저기서 $\alpha$ 가 알고리즘 속도 조절 parameter. step size 또는 learning rate(학습율,learning_rate).
이 방법의 문제점은 local_minimum 에 빠지는 것.
또 다른 문제점은 해에 근접할수록 $|\nabla f|$ 가 0에 가까워지기 때문에 수렴 속도가 느려진다는 것.
(darkpgmr)


[edit]

1. gradient descent에서 batch

배치,batch
// from https://velog.io/@arittung/DeepLearningStudyDay8#04-배치-사이즈-batch-size
{
batch는 전체 자료집합,dataset.
batch size는 단일 반복에서 gradient를 계산하는 데 사용하는 data의 총 개수.
mini batch는 1 batch size에 해당하는 dataset.

batch size가 너무
크면 - 느리고, 메모리부족 발생할 위험
작으면 - 너무 잦은 weight update - 훈련이 불안정해짐

(batch) gradient descent는 전체 dataset을 사용하므로 - 느리므로, 표본,sample을 무작위로 선택하는 대안 방법인
stochastic gradient descent (SGD)가 있음.
}

[edit]

2. CLEANUP




Links ko
http://www.aistudy.com/math/gradient_descent.htm 보면 역전파,backpropagation와 관련.
https://angeloyeo.github.io/2020/08/16/gradient_descent.html
aka : steepest_descent
대충, 최소가 되는 지점을 더듬어 찾아가는 방법?
안개로 주위가 보이지 않는 산에서 주위를 더듬으면서 낮은 곳으로 내려가는..
미분계수가 0인 지점을 찾으면 되지 않느냐? 라고 생각할 수 있는데, 근을 계산하기 어려운/불가능한 경우가 많아, 이 방법이 있는 것이다.
근데 global_minimum이 아닌 local_minimum에 빠져서 헤어나오지 못할 가능성이, 즉 optimum(최적해,optimal_solution < 해,solution, rel. 최적화,optimization)에 도달하지 못할 가능성이.. - greedy_algorithm과 비슷한 문제점

최적화,optimization 문제 해결방법의 하나?
다른 것은 라그랑주_곱셈자,Lagrange_multiplier... 비교?

[edit]

3. tmp bmks; 읽어서 흡수

이하 http://google.com/search?q=gradient descent 번역 검색결과 중
https://hyunw.kim/blog/2017/11/01/Optimization.html
{
순서대로
batch_gradient_descent BGD
stochastic_gradient_descent SGD
mini-batch_gradient_descent
momentum (운동량,momentum과 별도의 모멘텀,momentum 페이지를 만들까)
RMSProp
Adam ... Adagrad + Momentum ??
이상 글에서 다루는 키워드들만 간단히 나열하였음
}

[edit]

4. tmp bmks; del

3Blue1Brown series S3 E2
경사 하강, 신경 네트워크가 학습하는 방법 | 심층 학습, 2장
https://youtu.be/IHZwWFHWa-w

[edit]

5. Sub: 확률적기울기하강 stochastic gradient descent (SGD)

확률적기울기하강 stochastic_gradient_descent SGD
확률적경사하강, stochastic gradient descent (SGD)
tmp bmks ko
http://sanghyukchun.github.io/74/ 의 50% 정도의 "Stochastic Gradient Descent" 문단 참조.
대충, 신경망,neural_network의 weight_parameter_update할 때 모두 계산하면(full batch) 너무 비효율적이므로 SGD라는 확률적 방법을 쓴다는 얘기. 이 때 'mini batch'를 만든다는.


[edit]

6. Sub: adaptive gradient, adagrad - adaptive_gradient

[edit]

7. subgradient descent? - no, subgradient method.


볼록최적화 문제의 해법 중 하나. 반복법.

https://convex-optimization-for-all.github.io/contents/chapter08/ 를 보면 항상 하강하는 gradient descent와 달리 항상 하강하지는 않으므로 subgradient descent라고 부르지는 않는다 함.

번역?
subgradient not in kms ... https://www.kms.or.kr/mathdict/list.html?key=kname&keyword=subgradient
as of 2023-04-22.
subderivative를 wpko에선 '하방미분'이라 번역했던데 그럼 이건 '하방기울기' and 하방기울기법?
sub-를 보통 부-라고 번역하던데 그럼 이건 '부미분' and 부미분법? (??)

식은 gradient descent와 비슷하며 역시 반복,iteration법임.

projected_subgradient_method
{
projected subgradient method
번역?


rel.
subgradient
{
번역?



rel.
subderivative and subdifferential
{
번역?
'subd' not in kms ... https://www.kms.or.kr/mathdict/list.html?key=kname&keyword=subd
as of 2023-04-22

보통 sub-는 부- 로 번역하는데
wpko에선 '하방미분'이라 번역했음.


저건 볼록해석,convex_analysis 볼록함수,convex_function 쪽에서
미분,derivative 미분,differential 을 일반화/확장한 개념인듯. (qqq 미분가능하지 않은 (i.e. 미분가능성,differentiability 없는) 볼록함수,convex_function도 다루기 위해??)


}

}



[edit]

8. proximal gradient descent

[edit]

9. natural gradient descent

natural gradient descent
natural_gradient_descent

via
"자연스러운 경사하강법" 정보기하학과 머신러닝 (3): 거울 하강법 https://horizon.kias.re.kr/24237/ 앞부분


[edit]

10. rel. mirror descent

[edit]

11. tmp video en

Solve any equation using gradient descent - YouTube
https://www.youtube.com/watch?v=0kFydRfswU8



http://www.aistudy.com/math/gradient_descent.htm
ML에서 흔히 사용되는 gradient descent 는 크게 두가지 형태, 즉 batch 와 on-line 이 있다, 그 둘의 절충인 mini-batches가 있다
batch_gradient_descent
on-line_gradient_descent

GD의 types는 ibm에 의하면
batch
stochastic
mini-batch
https://www.ibm.com/topics/gradient-descent




이름에 대해
단변수일 경우 기울기,gradient를 정의할 수 없으므로, 도함수 하강(derivative descent)이라 명명하는 편이 나을지도.
"Especially, for the single-variable cases, probably I would say 'derivative descent algorithm' because we cannot define the gradient in this single-variable case." (KU정태수 ㄷㄱㄱ 14-2 18:00)

AKA 경사강하, 경사하강 (gradient descent)
AKA 경사하강법(gradient descent method/algorithm)
(기울기|경사)(하강|강하)(법)



Warning: fopen(./data/cache/backlinks/a/a1/a1f2f4212c00fc47079bb1bca94bbd61) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/f/f5/f5a2cc3a6142cd4fa20d89d7b0905672) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/f/f7/f76ba96518ace3fa7f0583b959e1c93e) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/a/ae/aeb99178f9956cea477e8dbba65cfd87) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/9/99/990a507a01cb4ba2eb80c7cde922e4d7) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/e/e1/e152d4bd92645c73becfeeb0c74448dd) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/6/6e/6ec58666e7574a9962b0c7a383c7d832) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/f/f0/f07b9022574030d2406783f4949aa3d9) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/a/a9/a930701b569e15b64c7c88a6857798a1) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/f/fb/fbe45970021fd6bdbefa1ffc29677196) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/9/90/902e4eb3e42b396f3a7da849c16e4a28) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/8/83/83e48817731dd899b2e9828ad585c35b) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/7/7c/7c613cc35261f98260a6e778f1999e97) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/8/8f/8fc5bfb1958943afb9d80b94a1eab8d7) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/d/d9/d98b6ef989a6fdbadf5f31bffb6a2e3b) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/8/87/871b53be33d13b699a401b5089d36f85) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/7/7d/7dfe583ff65f92f83087aa9389ed88f3) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/5/5c/5ce5d0322c95b0bc16980def3f9d77eb) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/a/a3/a33f7c37d6d2ed862085672488839264) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/9/96/963791dcf5007be3f76711d34cd1a4cf) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/c/c0/c057456b06f1e66d48970a047eaf9677) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/c/c7/c733d4f1b84358832b508411e9e1be1e) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/7/79/7963e5132fb9f5e957ba209c9993ea74) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/e/e9/e9bc30cb8a545928437970ff5e09b02a) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/7/70/70a9c75cef77857f6d1bba47a1374f67) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/d/dd/ddf3c0c38ffa70224c4332022a7a05f3) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/6/6a/6a16efc795aa011ee2758d99315c16ae) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/a/a4/a4fb53b8376f79520fd0d25bd0ad17c1) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/d/de/de0a9c8c9d6033686e5f34ee1c32688a) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/9/98/98e00234943bab1c7b30ab0423caca7c) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277

Warning: fopen(./data/cache/backlinks/d/d9/d93aaa6630ead365670fdbec58bf34af) [function.fopen]: failed to open stream: No such file or directory in /home1/tomoyo/public_html/123/lib/cache.text.php on line 277