python – Keras – 实现Rprop算法的问题

前端之家收集整理的这篇文章主要介绍了python – Keras – 实现Rprop算法的问题前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

我正在尝试为Keras(link)实现弹性反向传播优化器,但具有挑战性的部分是能够根据其相应的梯度是正,负还是零来对每个参数执行更新.我编写了下面的代码作为实现Rprop优化器的开始.但是,我似乎无法找到单独访问参数的方法.循环参数(如下面的代码所示)在每次迭代时返回p,g,g_old,s,wChangeOld,它们都是矩阵.

有没有办法可以迭代各个参数并更新它们?如果我可以根据其渐变的符号索引参数向量,它也会起作用.谢谢!

class Rprop(Optimizer):
    def __init__(self,init_step=0.01,**kwargs):
        super(Rprop,self).__init__(**kwargs)
        self.init_step = K.variable(init_step,name='init_step')
        self.iterations = K.variable(0.,name='iterations')

        self.posStep = 1.2
        self.negStep = 0.5
        self.minStep = 1e-6
        self.maxStep = 50.

    def get_updates(self,params,constraints,loss):
        grads = self.get_gradients(loss,params)
        self.updates = [K.update_add(self.iterations,1)]

        shapes = [K.get_variable_shape(p) for p in params]
        stepList = [K.ones(shape)*self.init_step  for shape in shapes]
        wChangeOldList = [K.zeros(shape) for shape in shapes]
        grads_old = [K.zeros(shape) for shape in shapes]

        self.weights = stepList + grads_old + wChangeOldList
        self.updates = []

        for p,wChangeOld in zip(params,grads,grads_old,stepList,wChangeOldList):
            change = K.sign(g * g_old)

            if change > 0:
                s_new = K.minimum(s * self.posStep,self.maxStep)
                wChange = s_new * K.sign(g)
                g_new = g

            elif change < 0:
                s_new = K.maximum(s * self.posStep,self.maxStep)
                wChange = - wChangeOld
                g_new = 0

            else:
                s_new = s
                wChange = s_new * K.sign(g)
                g_new = p

            self.updates.append(K.update(g_old,g_new))
            self.updates.append(K.update(wChangeOld,wChange))
            self.updates.append(K.update(s,s_new))

            new_p = p - wChange

            # Apply constraints
            if p in constraints:
                c = constraints[p]
                new_p = c(new_p)

            self.updates.append(K.update(p,new_p))
        return self.updates

    def get_config(self):
        config = {'init_step': float(K.get_value(self.init_step))}
        base_config = super(Rprop,self).get_config()
        return dict(list(base_config.items()) + list(config.items()))
最佳答案
我也在Keras寻找RProp算法并找到了这个问题.我冒昧地根据我的目的调整你的代码,现在把它发布回来.到目前为止它似乎工作得很好,但我没有广泛测试它.

免责声明:我对keras很新,但对theano(和块)有很多经验.此外,我只测试了theano作为后端,但没有张量流.

class RProp(Optimizer):
    def __init__(self,init_alpha=1e-3,scale_up=1.2,scale_down=0.5,min_alpha=1e-6,max_alpha=50.,**kwargs):
        super(RProp,self).__init__(**kwargs)
        self.init_alpha = K.variable(init_alpha,name='init_alpha')
        self.scale_up = K.variable(scale_up,name='scale_up')
        self.scale_down = K.variable(scale_down,name='scale_down')
        self.min_alpha = K.variable(min_alpha,name='min_alpha')
        self.max_alpha = K.variable(max_alpha,name='max_alpha')

    def get_updates(self,params)
        shapes = [K.get_variable_shape(p) for p in params]
        alphas = [K.variable(numpy.ones(shape) * self.init_alpha) for shape in shapes]
        old_grads = [K.zeros(shape) for shape in shapes]
        self.weights = alphas + old_grads
        self.updates = []

        for param,grad,old_grad,alpha in zip(params,old_grads,alphas):
            new_alpha = K.switch(
                K.greater(grad * old_grad,0),K.minimum(alpha * self.scale_up,self.max_alpha),K.maximum(alpha * self.scale_down,self.min_alpha)
            )
            new_param = param - K.sign(grad) * new_alpha
            # Apply constraints
            if param in constraints:
                c = constraints[param]
                new_param = c(new_param)
            self.updates.append(K.update(param,new_param))
            self.updates.append(K.update(alpha,new_alpha))
            self.updates.append(K.update(old_grad,grad))

        return self.updates

    def get_config(self):
        config = {
            'init_alpha': float(K.get_value(self.init_alpha)),'scale_up': float(K.get_value(self.scale_up)),'scale_down': float(K.get_value(self.scale_down)),'min_alpha': float(K.get_value(self.min_alpha)),'max_alpha': float(K.get_value(self.max_alpha)),}
        base_config = super(RProp,self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

重要笔记:

> RProp通常不包含在机器学习库中,原因如下:除非您使用全批量学习,否则它根本不起作用.全批量学习仅在您拥有小型训练集时才有用.
> Adam(Keras builtin)胜过这个RProp算法.也许是因为它就是这样,或者因为我弄错了:)

关于您的代码的一些注释(指您的原始变量名称):

> wChange从不在迭代中使用,因此您不需要将它们存储在永久变量中.
>改变> 0不会按照您的想法执行,因为更改是张量变量.你想要的是元素比较,使用K.switch()代替.
>您使用maxStep两次而不是另一次使用minStep.
>变化为零的情况可以忽略不计,因为这在实践中几乎不会发生.
> g_new = 0和g_new = p都是完全假的,应该是第一个if分支中的g_new = g.

原文链接:https://www.f2er.com/python/438738.html

猜你在找的Python相关文章