Skip to main content

Monitoring Smart Mover

Comments

27 comments

  • Jim Meyer

    Peter....  all great suggestions.  I am currently working on a major release and will try to implement as many as possible.

    1
  • Peter Gersmann

    Hi Jim,

    Thanks! Looking forward to it. Let me know if you need error messages, processes, named queries,...

    //Peter

    0
  • Pascal Reutener

    Jim Meyer Did you actually implement the suggested ideas? Could you please shed some light on this? Thank you!

    0
  • Jim Meyer

    Hi Pascal....

    Yes, some of them.  An error email is now in HTML format with any error lines in red.  The Elvis/Assets messages have been improved.

     

    0
  • Siebrand Mazeland

    Suppose that we would like to "tail" the logs generated by a SmartMover Service installed on Linux. How would we go about that?

    0
  • Jim Meyer

    Siebrand....  

    I would not advise opening the files that way for several reasons.  First the file is not just the log text, it includes other data (date-time, error flag, etc).... it also includes then entire un-purged history of the Process not just the last run.  Additionally you would need to make sure you do not lock the file as Mover may need to write to it.

    0
  • Siebrand Mazeland

    Thanks, Jim. Ergo: SmartMover does not allow accessing logs* in such a way they can be used in centralized logging?

    * Application/service logs as well as job logs.

    0
  • Jim Meyer

    There are situations where Mover will write to the operating system log but that is typically on startup or a crash.  The Process logs are all stored in text files as mentioned above in Mover's own preferences folders.

    0
  • Peter Gersmann

    Hi Jim,

    For Studio Server, Assets, and soon JournalDesigner we're consolidating logs into Datadog. Instead of having staff manually find and read red lines in mails, we have automated the monitoring of millions of log entries. The monitors find errors and create incidents in ServiceNow (our incident management platform). Severity and linking to business services is automated. It allows us to automate the incident handling to the point where our system is actively texting or calling support agents. This often cuts the resolution time by hours and in addition it reduces the cost.

    But it all begins with having a structured logfile that we can read or having the logs (or just the errors) written to the system log.

    0
  • Jim Meyer

    Hi Peter...

    I can certainly look into the possibility of integrating Mover's logging into Datadog.  Just to make sure we are on the same page here, is this what I should be looking at: https://docs.datadoghq.com/getting_started/integrations/

    Jim

     

    0
  • Siebrand Mazeland

    Thanks for looking into this, Jim. From my perspective it's not necessary to make the effort of integrating with Datadog specifically.

    When looking at log aggregation requirements specifically, it's important that at least a file (or more files with documented names) is available that can be tailed by a log relay client running on the instance the SmartMover Service is running on that is allowed to tail (non-blocking) log files in a (preferably) documented location. Additionally, it would be very useful to have the log entries be consistently formatted, ideally in JSON format. Having a "status" field will allow prioritizing log entries which is very helpful for triggering actions in the log aggregation platform, so less analysis of the actual messages is required.

    0
  • Peter Gersmann

    Hi Jim,

    Thanks! I agree with Siebrand that you don't need to create a special integration for Datadog. Structured logs that can be read is what we're looking for. We're ready to test and share our experiences with the community if other WoodWing users are interested.

    //Peter

    0
  • Jim Meyer

    Do you need all logs to be structured or just when there is an error?

    What if the error email that Mover already sends included a JSON enclosure with the same data as the email body but obviously more structured?

    Jim

     

    0
  • Siebrand Mazeland

    Do you need all logs to be structured or just when there is an error?

    All entries in a particular file should be formatted the same way, in my opinion.

    Example entry in WoodWing Assets:

    {"@timestamp":"2021-05-06T12:31:39.590Z","source_host":"ip-10-0-20-19.eu-west-1.compute.internal","file":"SourceFile","method":"apply","level":"INFO","line_number":"107","thread_name":"elvis-io-thread-2","@version":1,"logger_name":"com.woodwing.elvis.info.boot","message":"Initializing web application context","class":"com.ds.acm.bbh","mdc":{}}

    Example entry in WoodWing Studio:

    {"@timestamp":"2021-05-06T13:24:54.552Z","log.level":"ERROR","event.action":"mysql","host.hostname":"stu-a-usr-i-redacted","client.ip":"redacted","package.name":"WoodWing Studio Server","package.version":"10.12.0 Build 333","url.original":"https:\/\/redacted\/server\/admin\/serverplugins.php","service.name":"","message":"MySQL: 1227:Access denied; you need (at least one of) the RELOAD privilege(s) for this operation"}

     

    What if the error email that Mover already sends included a JSON enclosure with the same data as the email body but obviously more structured?

    I don't think email has a place in centralized logging. It comes across as atomic and overly complex, not suited for in time and a large volume.

    0
  • Jim Meyer

    It is certainly possible to change Mover so that the logs are all stored in JSON files...  but I would have to see how that might impact performance of the product overall.

    Currently as a Process runs the code simply appends each log entry onto the end of a plain text file...  which is very fast. Those files can get very large.  Converting the files to JSON and appending new entries would add a lot more overhead. 

    Each Process has its own log file which is located in folder that is named the same as the Process itself.  There is also a "master" log file that includes an entry for each time a Process runs.

    The Process log files do have a "pipe" delimited structure:

    Date | Time | log text | error bool (t/f) | logID

    Can you work with that format?

    Jim

    0
  • Siebrand Mazeland

    The Process log files do have a "pipe" delimited structure:

    Date | Time | log text | error bool (t/f) | logID

    Can you work with that format?

    Does it work consistently, without potentially accidentally adding new fields when a pipe is present in the text of a log file? Fields would have to be quoted (with quotes escaped in contents), I think, if there is a possibility that pipes are present.

    I'm writing the above because it's my experience that all these log processors generally have capabilities to transform and map fields to a given structure.

    0
  • Jim Meyer

    If there can be a pipe in the log text it is a bug that can be fixed.

    0
  • Siebrand Mazeland

    If there can be a pipe in the log text it is a bug that can be fixed.

    👍🏽

    0
  • Sebastian Nafroth

    If possible, I'd like to see the message ("log text") in the last coloumn. This makes it a bit more human readable - if ever needed.

    0
  • Jim Meyer

    Changing the order of the fields is possible but would require converting all the existing data on installation of the new version.  This reduce backwards compatibility.

    The raw log files were never meant to be human readable.

    0
  • Siebrand Mazeland

    Jim, I've looked at the SmartMover log files in Linux a little closer. All those log files consist of a single line, and the records are apparently separated by ^M, like this:

    05/07/2021|00:00 AM|Logs Purged|f|3703190401^M05/07/2021|00:04 AM|Archiv Assets SmartMover|f|3703190699^M...

    I'm pretty sure it will prove to be impossible to tail these files, as there is never a newline...

    0
  • Jim Meyer

    I believe that "^M" is a visual representation of a carriage return.  When I open a Log file using a plain text editor on my Linux machine I see something like:

    12/28/2020|09:37 AM|Process started by Manual (09:37:41 AM)|f|3691993061
    12/28/2020|09:37 AM|Running Task 'Copy Files' (09:37:41 AM)|f|3691993061
    12/28/2020|09:37 AM|Copy Files (09:37:41 AM)|f|3691993061

    ...  and when I hexdump the file I see a hex "0D" at the end of each entry...  which is what I expect.

    0
  • Siebrand Mazeland

    From what I can tell, the carriage return is there in SmartMover log files, but the newline character appears to be missing. I've confirmed this for SmartMover logs on Linux at two different customers. In "vi" the files named "Log" only have a single line.

    0
  • Jim Meyer

    Yes, each line in the Log file ends with just a carriage return.  This was by design...  Mover runs on 3 operating systems that have different line ending standards.  The logs files were never intended to be accessed outside of the app and I wanted consistency between the 3.

    Can you copy the file and replace the returns with line feeds?

    0
  • Siebrand Mazeland

    Can you copy the file and replace the returns with line feeds?

    Yes, but then the file cannot be tailed anymore. Copying a file every 10 seconds, and then diffing it to determine the differences compared to the version 10 seconds before that appears to be sub-optimal.

    0
  • Jim Meyer

    All new entries are appended to the bottom of the file so you only need to look at the new lines or the number of bytes added since the previous check....  (except when the Logs are purged nightly and the file sized is reduced).  I believe that "tail" with the -c option can do that.

    But I am concerned about how this would work.  As the Task runs, Mover opens the file with write access, appends the entry and then closes it.  You need to make sure that you do not prevent Mover from doing that at any time.  The code assumes exclusive use of the file.

    0
  • Siebrand Mazeland

    All new entries are appended to the bottom of the file

    Yes. But as there are no newlines, I fear that the agents will not be able to separate events. I'll test this sometime soon, and will let you know. Newlines matter...

    But I am concerned about how this would work. As the Task runs, Mover opens the file with write access, appends the entry and then closes it. You need to make sure that you do not prevent Mover from doing that at any time

    Sure. That's the whole idea behind "tail"-ing: showing what was added to a file without interfering with its contents or changes that are made to it.

    0

Please sign in to leave a comment.

Post ID: 360048116531

Created at: 2019-07-30 11:11:00 UTC

Updated at: 2021-05-10 16:47:40 UTC

Edited at:

Comment count: 27

Follower count: 6

Vote sum: 3

Vote count: 3

Status: none