プロジェクト

全般

プロフィール

Vote #77529

完了

Error with Fetch commits with Mercurial repository when log has invalid char

Admin Redmine さんが3年以上前に追加. 3年以上前に更新.

ステータス:
Closed
優先度:
通常
担当者:
-
カテゴリ:
SCM_3
対象バージョン:
開始日:
2022/05/09
期日:
進捗率:

0%

予定工数:
category_id:
3
version_id:
152
issue_org_id:
23055
author_id:
153186
assigned_to_id:
11192
comments:
28
status_id:
5
tracker_id:
1
plus1:
0
affected_version:
closed_on:
affected_version_id:
109
ステータス-->[Closed]

説明

Error with Fetch commits automatically with mercurial repository

When I add a mercurial repository, redmine start to scan and add to Mysql. This take several minutes. When redmine finish to index .Redmine show a error page. In production.log show this message. It happen with a huge repository about 70335 commits

production.log

Started GET "/projects/iset/repository" for 192.168.1.164 at 2016-06-13 10:56:04 +0000
Processing by RepositoriesController#show as HTML
  Parameters: {"id"=>"iset"}
  Current user: admin (id=1)
Completed 500 Internal Server Error in 2039ms (ActiveRecord: 4.8ms)

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:194:in `each_revision'
  app/models/repository/mercurial.rb:187:in `block in fetch_changesets'
  app/models/repository/mercurial.rb:186:in `step'
  app/models/repository/mercurial.rb:186:in `fetch_changesets'
  app/controllers/repositories_controller.rb:114:in `show'
  lib/redmine/sudo_mode.rb:63:in `sudo_mode'

If you exec a manual auto_fetch to refresh data with rake -f Rakefile RAIL_ENV=production redmine:fetch_changesets --trace
Console shows:

/usr/local/bundle/gems/htmlentities-4.3.1/lib/htmlentities/mappings/expanded.rb:465: warning: duplicated key at line 466 ignored: "inodot"
** Invoke redmine:fetch_changesets (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:fetch_changesets
rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass
/usr/src/redmine/lib/redmine/scm/adapters/mercurial_adapter.rb:194:in `each_revision'
/usr/src/redmine/app/models/repository/mercurial.rb:187:in `block in fetch_changesets'
/usr/src/redmine/app/models/repository/mercurial.rb:186:in `step'
/usr/src/redmine/app/models/repository/mercurial.rb:186:in `fetch_changesets'
/usr/src/redmine/app/models/repository.rb:344:in `block (2 levels) in fetch_changesets'
/usr/local/bundle/gems/activerecord-4.2.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/local/bundle/gems/activerecord-4.2.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/src/redmine/app/models/repository.rb:342:in `block in fetch_changesets'
/usr/local/bundle/gems/activerecord-4.2.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/local/bundle/gems/activerecord-4.2.5/lib/active_record/relation/delegation.rb:46:in `each'
/usr/src/redmine/app/models/repository.rb:341:in `fetch_changesets'
/usr/src/redmine/lib/tasks/redmine.rake:47:in `block (2 levels) in '
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:240:in `call'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/usr/local/bundle/gems/rake-10.5.0/bin/rake:33:in `'
/usr/local/bundle/bin/rake:16:in `load'
/usr/local/bundle/bin/rake:16:in `
' Tasks: TOP => redmine:fetch_changesets
  Redmine version                3.1.3.stable
  Ruby version                   2.2.4-p230 (2015-12-16) [x86_64-linux-gnu]
  Rails version                  4.2.5
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.8.13
  Mercurial                      3.1.2
  Git                            2.1.4
  Bazaar                         2.7.0
  Filesystem                     
Redmine plugins:
  no plugin installed

journals

What output of scm_stderr_log_file?
source:tags/3.2.3/config/configuration.yml.example#L138
--------------------------------------------------------------------------------
I haven't any exit on scm_stderr_log_file.

But in production.log when do the first scan logged this:

<pre>
Changeset Load (0.8ms) SELECT `changesets`.* FROM `changesets` WHERE `changesets`.`repository_id` = 11 AND `changesets`.`scmid` = '6e892cac598ab919f853ca54c7745e783b70dcb8' ORDER BY changesets.id DESC LIMIT 1
Changeset Exists (0.6ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`revision` = BINARY '10555' AND `changesets`.`id` != 37510 AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.5ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`scmid` = BINARY '6e892cac598ab919f853ca54c7745e783b70dcb8' AND `changesets`.`id` != 37510 AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.4ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`revision` = BINARY '10598' AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.4ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`scmid` = BINARY '6289f36f5adf23615f6fc2f49abd5828beffb29b' AND `changesets`.`repository_id` = 11) LIMIT 1
SQL (0.5ms) INSERT INTO `changesets` (`repository_id`, `revision`, `scmid`, `committer`, `commit_date`, `committed_on`, `comments`, `user_id`) VALUES (11, '10598', '6289f36f5adf23615f6fc2f49abd5828beffb29b', 'pepe<eduardo@pepe.es>', '2011-07-15', '2011-07-15 07:42:42', 'TextStyle ahora senyala el estilo del caracter donde esta el cursor.', 6)
SQL (0.3ms) INSERT INTO `changeset_parents` (`parent_id`, `changeset_id`) VALUES (37510, 37553)
SQL (0.3ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37553, 'M', '/core/exe/edtext.cpp')
SQL (0.3ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37553, 'M', '/core/ipf/ctext.cpp')
SQL (0.3ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37553, 'M', '/core/ipf/ctext.h')
(34.2ms) COMMIT
(0.2ms) BEGIN
Changeset Load (0.7ms) SELECT `changesets`.* FROM `changesets` WHERE `changesets`.`repository_id` = 11 AND `changesets`.`scmid` = '6289f36f5adf23615f6fc2f49abd5828beffb29b' ORDER BY changesets.id DESC LIMIT 1
Changeset Exists (0.6ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`revision` = BINARY '10598' AND `changesets`.`id` != 37553 AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.5ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`scmid` = BINARY '6289f36f5adf23615f6fc2f49abd5828beffb29b' AND `changesets`.`id` != 37553 AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.4ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`revision` = BINARY '10599' AND `changesets`.`repository_id` = 11) LIMIT 1
Changeset Exists (0.4ms) SELECT 1 AS one FROM `changesets` WHERE (`changesets`.`scmid` = BINARY '2babe5c449c6274f052a8bda9637c247ea210e23' AND `changesets`.`repository_id` = 11) LIMIT 1
SQL (0.5ms) INSERT INTO `changesets` (`repository_id`, `revision`, `scmid`, `committer`, `commit_date`, `committed_on`, `comments`, `user_id`) VALUES (11, '10599', '2babe5c449c6274f052a8bda9637c247ea210e23', 'pepe<eduardo@pepe.es>', '2011-07-19', '2011-07-19 16:04:02', 'Soporte de estilos usando la seleccion de texto.', 6)
SQL (0.3ms) INSERT INTO `changeset_parents` (`parent_id`, `changeset_id`) VALUES (37553, 37554)
SQL (0.3ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37554, 'M', '/core/exe/edtext.cpp')
SQL (0.3ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37554, 'M', '/core/ipf/ctext.cpp')
SQL (0.4ms) INSERT INTO `changes` (`changeset_id`, `action`, `path`) VALUES (37554, 'M', '/core/ipf/ctext.h')
(26.2ms) COMMIT
Shelling out: 'hg' '-R' '/home/share/mirror/hg/all' '--encoding' 'utf-8' '--config' 'extensions.redminehelper=/usr/src/redmine/lib/redmine/scm/adapters/mercurial/redminehelper.py' '--config' 'diff.git=false' 'log' '--debug' '-C' '--style' '/usr/src/redmine/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl' '-r' '10600:10699'
Completed 500 Internal Server Error in 893414ms (ActiveRecord: 465576.7ms)

NoMethodError (undefined method `[]' for nil:NilClass):
lib/redmine/scm/adapters/mercurial_adapter.rb:194:in `each_revision'
app/models/repository/mercurial.rb:187:in `block in fetch_changesets'
app/models/repository/mercurial.rb:186:in `step'
app/models/repository/mercurial.rb:186:in `fetch_changesets'
app/controllers/repositories_controller.rb:114:in `show'
lib/redmine/sudo_mode.rb:63:in `sudo_mode'
</pre>
--------------------------------------------------------------------------------
mole mole wrote:
> I haven't any exit on scm_stderr_log_file.

How about "#{Rails.env}.scm.stderr.log"?
source:tags/3.2.3/lib/redmine/scm/adapters/abstract_adapter.rb#L221

>
> But in production.log when do the first scan logged this:
> ...
> 'hg' '-R' '/home/share/mirror/hg/all' '--encoding' 'utf-8' '--config' 'extensions.redminehelper=/usr/src/redmine/lib/redmine/scm/adapters/mercurial/redminehelper.py' '--config' 'diff.git=false' 'log' '--debug' '-C' '--style' '/usr/src/redmine/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl' '-r' '10600:10699'

What output of above "hg" on command line?

--------------------------------------------------------------------------------
I haven't any exit on scm_stderr_log_file. -> I try to say: I haven't any error related with this issue. Only have this message:
not trusting file /home/share/mirror/hg/all/.hg/hgrc from untrusted user root, group www-data

It can be solve with adding in hgrc.
<pre>
[trusted]
users =

</pre>

hg in command line:

<pre>
Mercurial Distributed SCM

basic commands:

add add the specified files on the next commit
annotate show changeset information by line for each file
clone make a copy of an existing repository
commit commit the specified files or all outstanding changes
diff diff repository (or selected files)
export dump the header and diffs for one or more changesets
forget forget the specified files on the next commit
init create a new repository in the given directory
log show revision history of entire repository or files
merge merge working directory with another revision
pull pull changes from the specified source
push push changes to the specified destination
remove remove the specified files on the next commit
serve start stand-alone webserver
status show changed files in the working directory
summary summarize working directory state
update update working directory (or switch revisions)

use "hg help" for the full list of commands or "hg -v" for details

</pre>
--------------------------------------------------------------------------------
What is output of following command?
'hg' '-R' '/home/share/mirror/hg/all' '--encoding' 'utf-8' '--config' 'extensions.redminehelper=/usr/src/redmine/lib/redmine/scm/adapters/mercurial/redminehelper.py' '--config' 'diff.git=false' 'log' '--debug' '-C' '--style' '/usr/src/redmine/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl' '-r' '10600:10699'
--------------------------------------------------------------------------------
The command 'hg' '-R' '/home/share/mirror/hg/all' '--encoding' 'utf-8' '--config' 'extensions.redminehelper=/usr/src/redmine/lib/redmine/scm/adapters/mercurial/redminehelper.py' '--config' 'diff.git=false' 'log' '--debug' '-C' '--style' '/usr/src/redmine/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl' '-r' '10600:10699'

Can be executed without errors. Attach output.
--------------------------------------------------------------------------------
What is output?
<pre><code class="sql">
select revision from changesets where id =
(select max(id) from changesets where repository_id = 11)
</code></pre>

--------------------------------------------------------------------------------
revision
10599
--------------------------------------------------------------------------------
Try reduce FETCH_AT_ONCE.
source:tags/3.1.3/app/models/repository/mercurial.rb#L30

I think a revision from 10600 to 10699 has problem.

--------------------------------------------------------------------------------
I think your revision has problem.
* no date
* no commit log

If you find this revision, try this patch.
<pre><code class="diff">
diff --git a/lib/redmine/scm/adapters/mercurial_adapter.rb b/lib/redmine/scm/adapters/mercurial_adapter.rb
--- a/lib/redmine/scm/adapters/mercurial_adapter.rb
+++ b/lib/redmine/scm/adapters/mercurial_adapter.rb
@@ -212,8 +212,8 @@ module Redmine
yield Revision.new(:revision => le['revision'],
:scmid => le['node'],
:author => (le['author']['__content__'] rescue ''),
- :time => Time.parse(le['date']['__content__']),
- :message => le['msg']['__content__'],
+ :time => (Time.parse(le['date']['__content__']) rescue Time.at(0)),
+ :message => (le['msg']['__content__'] rescue ''),
:paths => paths,
:parents => parents_ary)
end

</code></pre>
--------------------------------------------------------------------------------
It didn't works. Same result.

Maybe a wrong code of EOL?. I see in sublime editor a DC3 code.

--------------------------------------------------------------------------------
I thing that double *!!* can break it.

The message in mercurial is:

<pre>
-dimensiones de los geosets de m3Font funcionando bien con shader.!!
</pre>

and the output of script have a DC3 code when in original have a *!!*:

<pre>
<msg>-dimensiones de los geosets de m3Font funcionando bien con shader.</msg>
</pre>

--------------------------------------------------------------------------------
Try upgrading Mercurial. Latest version is 3.8.3.
--------------------------------------------------------------------------------
It didn't works. Same result.

--------------------------------------------------------------------------------
Please report to Mercurial BTS.
Broken XML is Mercurial bug.
https://bz.mercurial-scm.org/
--------------------------------------------------------------------------------
The character is ^S I try to reproduce in small repo
--------------------------------------------------------------------------------
You can reproduce this issue with this repo.
--------------------------------------------------------------------------------
It is obviously Mercurial bug.
Please report to Mercurial BTS.
<pre>
$ LANG=C /WEB-DOWN/hg-repo/hg-crew/hg log --version
Mercurial Distributed SCM (version 3.8.3)
(see https://mercurial-scm.org for more information)

Copyright (C) 2005-2016 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ /WEB-DOWN/hg-repo/hg-crew/hg log --debug --style /REDMINE/MY-REDMINE/test-trunk/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl > hg.xml

$ xmllint hg.xml
hg.xml:10: parser error : PCDATA invalid Char value 19
<msg>-Test</msg>
^
hg.xml:17: parser error : Premature end of data in tag log line 2

</pre>
--------------------------------------------------------------------------------
Sorry, we need add "</log>".

<pre>
$ /WEB-DOWN/hg-repo/hg-crew/hg log --debug --style /REDMINE/MY-REDMINE/test-trunk/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl > hg.xml

$ echo "</log>" >> hg.xml

$ xmllint hg.xml
hg.xml:23: parser error : PCDATA invalid Char value 19
<msg>-Test</msg>
^
</pre>
--------------------------------------------------------------------------------
Using Mercurial xml template generate a correct xml

<pre>
$ hg log --debug --style=xml > hg.xml

$ xmllint hg.xml
</pre>

It runs correctly

PS: --style is a deprecated parameter in mercurial
--------------------------------------------------------------------------------
I changed in hg-template-1.0.tmpl {desc|escape} by {desc|revscape} and xmllint runs correctly...
--------------------------------------------------------------------------------
Sorry, its probably we must use xmlescape

{desc|xmlescape}

I see this in this file sample of mercurial

<pre>
/usr/share/mercurial/templates/map-cmdline.xml
</pre>
--------------------------------------------------------------------------------
Attach hg-template-1.0.tmpl modified
--------------------------------------------------------------------------------
Sorry, is this file
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Thanks for your help.
--------------------------------------------------------------------------------
I can also confirm this. Any workaround?

Environment:
Redmine version 3.2.0.stable
Ruby version 2.3.1-p112 (2016-04-26) [x86_64-linux-gnu]
Rails version 4.2.5
Environment production
Database adapter PostgreSQL
SCM:
Mercurial 3.7.3
--------------------------------------------------------------------------------
I choose 'urlescape' because 'paths' uses 'urlescape' and decodes it.
source:trunk/lib/redmine/scm/adapters/mercurial_adapter.rb@19606#L204
--------------------------------------------------------------------------------

Admin Redmine さんが3年以上前に更新

  • カテゴリSCM_3 にセット
  • 対象バージョン4.2.0_152 にセット

他の形式にエクスポート: Atom PDF

いいね!0
いいね!0