Accessing emails on Office 365 via OAuth

We’re creating an app that accesses email-accounts via imap to import mails into our app to document what is happening in a project. The same with sending, we can send information from our system via the mailserver the customer uses.

We use the mailpro plugin to access the accounts and folders, and the user only has to enter infos about the server, like name, port and his username and password.

Beginning October 2022 Microsoft will remove the option to login with name and password (”basic authentication”) and will only allow users to login with “modern authentication” (OAuth).

Sending mail via SMTP can still be used, so customers with scanners / multi-function-devices that send scans via email will still work, but this has to be activated manually.

Microsoft will not offer the “application password” system like apple or google do to enable access to the account.

So we (and everyone else that uses imap with Office 365) will have to switch to the new authentification-flow described here: [Authenticate an IMAP, POP or SMTP connection using OAuth | Microsoft Learn](https://learn.microsoft.com/en-us/excha … cess-token)

An example with a postman-demonstration is found here: [OAuth 2.0 client credentials flow on the Microsoft identity platform - Microsoft identity platform | Microsoft Learn](https://learn.microsoft.com/en-us/azure … grant-flow)

So basically we are doing this:

we registered an app on [portal.azure.com](http://portal.azure.com) after creating a free account / using our existing Office 365 - Account

We got an app-id and a secret for accessing the app

we adapted the workflow from the postman-flow linked above for Servoy:

/**
 * generated secret
 * temporary key, works till 2022-10-01 
 * @type {String}
 *
 * @properties={typeid:35,uuid:"6534DE9D-2D01-4D93-BC45-BE8D011CF357"}
 */
var clientSecret = 'ZWt8Q~VS~HN_dE.Qrrmq4ymf-X5c0AesQBNgIcSf';

/**
 * client-id, generated by Azure unpon registration
 * @type {String}
 *
 * @properties={typeid:35,uuid:"F4FC6CD6-0814-4F12-A5F8-262B51FDF5DF"}
 */
var clientId = '1c62cf99-7745-4589-8f89-55f084f4d2a4';
/**
 * @type {String}
 *
 * @properties={typeid:35,uuid:"8B665560-A9C8-4385-8EEE-53D59E4E6846"}
 */
var state = 'SecretSauce22';

/**
 * @type {String}
 *
 * @properties={typeid:35,uuid:"2DF578C4-E259-46DF-83E1-A216A68567D5"}
 */

var scope = 'openid offline_access https://graph.microsoft.com/mail.read';

/**
 * @type {String}
 *
 * @properties={typeid:35,uuid:"2E35486A-93C9-4C24-86A9-05EA139FFD7C"}
 */
var redirectUrl = 'http://localhost:8183/solutions/office365_test/m/onO365Authorize';

/**
 * @type {String}
 *
 * @properties={typeid:35,uuid:"BE03A4A2-851D-4F19-AB0F-A19B21BDA5A6"}
 */
var user_email = 'Robert.Edelmann@BauProCheck.de';
	
/**
 * Callback function, receives informations after users accepts login via OAuth
 * has to be in scopes.globals of the main solution
 * information should be transferred to the correct scope via
 * scopes.office365.onO365Authorize(a,args);
 * @param a
 * @param args
 *
 * @properties={typeid:24,uuid:"D824F716-F42F-4525-8314-CBD408F3A547"}
 */
function onO365Authorize (a, args) {
	if (args && args.hasOwnProperty('code') && args['code']) {
//		application.output('found code: ' + args['code']);
		getImapLoginToken(args['code']);
	}
}

/**
 * the function that starts it all; 
 * @properties={typeid:24,uuid:"004FDE7C-780E-4DAD-961C-BD1B30365919"}
 */
function authO365_idToken() {
	var authURL = 'https://login.microsoftonline.com/common/oauth2/v2.0/authorize?';
	authURL += 'client_id=' + clientId;
	authURL += '&response_type=code';
	authURL += '&redirect_uri=' + redirectUrl;
	authURL += '&response_mode=query';
	authURL += '&scope=' + scope;
	authURL += '&state='+state;
	application.output('URL: ' + authURL);
	application.showURL(authURL, '_blank');
}

/**
 * @param {String} code code from authorization
 * @properties={typeid:24,uuid:"87743126-4782-4CCD-AEEE-B5376C5FB1B2"}
 */
function getImapLoginToken(code) {
	if (!code) {
		return;
	}
	var httpClient = plugins.http.createNewHttpClient();
	var request = httpClient.createPostRequest('https://login.microsoftonline.com/common/oauth2/v2.0/token');
	request.addHeader('Content-Type', 'application/x-www-form-urlencoded');
	var bodyContent = 'client_id='+clientId;
	bodyContent += '&scope=https://outlook.office365.com/.default';
	bodyContent += '&redirect_uri='+redirectUrl;
	bodyContent += '&grant_type=authorization_code';
	bodyContent += '&client_secret='+clientSecret;
	bodyContent += '&code=' + code;
	
	request.setBodyContent(bodyContent);
	var response = request.executeRequest();
	var statusCode = response.getStatusCode()
	if (statusCode != 200) {
		application.output('Error processing request, Statuscode ' + statusCode.toString() + '\n' + response.getResponseBody());
		return;
	} else {
//		application.output(response.getResponseBody());
		/** @type {{token_type: String, scope: String, expires_in: String, ext_expires_in: String, access_token: String, refresh_token: String, id_token: String}} */
		var imapLoginObject = JSON.parse(response.getResponseBody());
		if (imapLoginObject && imapLoginObject.hasOwnProperty('access_token') && imapLoginObject.access_token) {
			getImapInbox(imapLoginObject.access_token);
		}
	}
}

/**
 * uses the access_token to authenticate
 * @param {String} [accessToken]
 *
 * @properties={typeid:24,uuid:"94418998-58D5-48A5-A448-A4CCC45BBD39"}
 */
function getImapInbox(accessToken) {
	if (!accessToken) {
		return;
	}
	var imapAccount = plugins.MailPro.ImapAccount('emailaccount', 'outlook.office365.com', user_email, accessToken);
	imapAccount.port = 993
	var props = {
		"mail.imap.fetchsize": java.lang.Integer.parseInt('1048576'),
		"mail.imaps.fetchsize": java.lang.Integer.parseInt('1048576'),
		"mail.imap.connectionpoolsize": "10",
		"mail.imaps.connectionpoolsize": "10",
		'mail.imaps.starttls.enable': true,
		'mail.imap.starttls.enable': true,
		"mail.imap.ssl.enable": true,
		"mail.imaps.ssl.enable": true,
		"mail.imap.auth.mechanisms": "XOAUTH2",
		"mail.imap.auth.plain.disable": true,
		"mail.imaps.auth.mechanisms": "XOAUTH2",
		"mail.imaps.auth.plain.disable": true
	};

	var rootFolder = imapAccount.connect(props);
	if (!rootFolder || !imapAccount.connected) {
		if (imapAccount.getLastError()) {
			throw imapAccount.getLastError();
		} else {
			return;
		}
	}
	var folder = imapAccount.getRootFolder()
	var subFoldees = folder.getSubfolders();
	for (var iFolders = 0; iFolders < subFoldees.length; iFolders++) {
		try {
			application.output(subFoldees[iFolders].name + ' -> ' + subFoldees[iFolders].getMessageCount().toString());
		} catch (e) {
			application.output('Error Accessing Folder: ' + e.name + ' -> ' + e.message + '\n' + e.stack,LOGGINGLEVEL.ERROR)
			break;
		}
	}
}

The code has to be in the globals scope of a solution named office365_test for the callback to work.

The project can be found on [GitHub - RobertEdelmann1974/office365_test](https://github.com/RobertEdelmann1974/office356_test)

This works somehow, but the the accessToken for imap only has a very limited duration, after about 2 minutes we are no longer authenticated.

We have heard from other users who can access imap longer, but not really for more than 1 oder 2 hours.

Does anyone else have information about this workflow, if and where I am wrong or any other way, apart from going the route of using graph instead of imap when accessing emails?

Hi,

we’ve also problems with the upcoming changes from Microsoft with “Modern auth”.
Like Robert, we also access Office365 emails via IMAP in our CRM solution. We do this on Servoy Server with a headless client / batchprocessor client.

Until now we connected the IMAP Account with the username/password. But from October 2022 Microsoft only allows OAuth 2.0 access to connect to IMAP server.

To support the Microsoft OAUTH flow we’ve already created an Microsoft Azure account and created an app to get the needed OAUTH data:

  • clientId
  • Auth URL
  • Token URL

In Servoy (SmartClient with Servoy 7) we use the svyOAuth module from https://github.com/Servoy/svyOAuth to manage the authorization and token workflow to get new access tokens.
This part already works. We successfully get OAuth tokens which are the new “passwords” to connect to the IMAP account.
Our code looks like this:

  1. Authorization:
function authorize(userName, scopeArray){
	auth = scopes.svyOAuth.createAuthorizationRequest()
		.setAuthServerURL(AUTH_URI)
		.setClientID(clientId)
		.setTokenServerURL(TOKEN_URI);
	if(scopeArray){
		for(var i in scopeArray){
			auth.addScope(scopeArray[i]);
		}
	}
	client = auth.execute(userName);
	return client;
}

var credentials = authorizedClient.getCredential();
if (credentials) {
   var refreshToken = credentials.getRefreshToken();
   var accessToken = credentials.getAccessToken();
}
  1. Get access token from refresh token:
var url = TOKEN_URI; // custom TOKEN URI from Azure portal 
var data = 'client_id=' + clientId + '&refresh_token=' + refreshToken + '&grant_type=refresh_token';

var vHttpClient = plugins.http.createNewHttpClient();
var vPostReq = vHttpClient.createPostRequest(url);
vPostReq.addHeader("Content-Type", "application/x-www-form-urlencoded");
vPostReq.setBodyContent(data);

var vResponse = vPostReq.executeRequest();
var vResponseBody = vResponse.getResponseBody();

if (vResponse.getStatusCode() == 200) {
	var vBody = JSON.parse(vResponseBody);
	accessToken = vBody.access_token;
}

We use the following scopes:

After getting the access token we also use the MailPro plugin from Patrick Ruhsert which was built for “normal” username/password workflows without OAuth and without token expiration:

var props = {
		"mail.imap.fetchsize": java.lang.Integer.parseInt('1048576'),
		"mail.imaps.fetchsize": java.lang.Integer.parseInt('1048576'),
		"mail.imap.connectionpoolsize": "10",
		"mail.imaps.connectionpoolsize": "10",
		"mail.imap.ssl.enable": "true",
		"mail.imap.auth.mechanisms": "XOAUTH2"
		"mail.imaps.ssl.enable": "true"
		"mail.imaps.auth.mechanisms": "XOAUTH2"
	};
	
var account = plugins.MailPro.ImapAccount(<name>, "outlook.office365.com", "<emailaddress>, null);
account.useSSL = true;
account.port = 993;

var imapRootFolder = account.connect("outlook.office365.com", "<emailaddress>", accessToken, props);

The connection to the IMAP account also works.
After the connection is established we register newMail- and changedMail-Listeners:

var imapFolder = account.getFolder("INBOX");
imapFolder.setNewMailCallbackMethod(globals.mkor_nachrichtNeu, vArgs, 4);
imapFolder.setChangedMailCallbackMethod(globals.mkor_nachrichtGeaendert, vArgs);

All this works like expected - but only until the access token is expired, which happens after about one hour.

I’ve already tried several workarounds, but nothing worked:
In all cases i created a new access token 10 min before the first access token expires and

  • set the new token in account.password
  • disconnect the account first and call account.connect() with the new token
  • disconnect the account, created a complete new plugins.MailPro.ImapAccount with a new name

The only “workaround” is to close the Servoy headless client and start the batchprocessor again.
But that can not be the solution.

After access token expiration the new/changedListeners fire errors:

MailPro.ChangedMailListener ERROR: changedMailListener could not open folder INBOX. Retrying in 1 minute…
MailPro.MsgCounterListener WARN: MsgCounterListener could not open folder INBOX. Retrying in 60 seconds…
MailPro.ChangedMailListener ERROR: changedMailListener error receiving new messages from folder + INBOX: javax.mail.FolderClosedException: failed to create new store connection

Anybody who has also customers accessing their email accounts via IMAP?

Thanks for your support!
Alex

Hello,

my name is Dmitrij Wanscheid and I am a software developer at HV-Office.

We run the MailPro plugin in several projects, also at customers.
After Microsoft announced that they will disable Basic authentication from October 2022 we had to react as well. Both software at the customer and the internal software at our company was affected.

For this we had to do research. We are currently already running OAuth2 (also in the production system). MailPro has thus been completely replaced and it works perfectly. For this we use the OnBoard plugin from Servoy.

The advantage of this method is that we can access the complete Microsoft API. Thus, not only the email is affected but it also offers the possibility to synchronize Outlook appointments, which will soon be implemented in our product as well. It is also planned to replace SMTP. So you will get the complete service with only one log-in.

Hi d.wanschied!

Thanks for your response!

Is it right that you use the onboard plugins.mail from Servoy to connect your office365 imap accounts with OAuth2 instead of the MailPro plugin?

Or do you complety use the Microsoft Graph API to do the “mail things” without using IMAP at all?

Thanks!

LXS:
Hi d.wanschied!

Thanks for your response!

Is it right that you use the onboard plugins.mail from Servoy to connect your office365 imap accounts with OAuth2 instead of the MailPro plugin?

Or do you complety use the Microsoft Graph API to do the “mail things” without using IMAP at all?

Thanks!

Hi LXS,

neither. We don’t use MailPro or mail plugin anymore.

Instead, we use the “plugins.oauth” plugin for authentication. With this we use the complete Microsoft Graph.
Not only emails can be retrieved here, but also many other things. Outlook appointments, user data of the Microsoft account etc…

So you get everything, only with one log-in.

Ah, i understand!
Do you use polling to get the latest emails or is there a push notification when new emails arrive?
How much effort was it to implement Microsoft Graph for email?

Do you use polling to get the latest emails or is there a push notification when new emails arrive?

For this purpose, a batch process runs that retrieves the e-mails at a certain interval. Microsoft offers various filter options via the API.
This way we keep the response as small as possible to prevent a timeout. Thus, we fetch only the most necessary data that we need for our functions.

How much effort was it to implement Microsoft Graph for email?

Since I already had a lot of experience in API, with projects before, the development and implementation in existing code had taken only about a week. With tests it was about 2-3 weeks.

d.wanscheid:
neither. We don’t use MailPro or mail plugin anymore.

Although I didn’t mention it as a problem, do you send mails via Graph or do you still use smtp for that part?

robert.edelmann:

d.wanscheid:
neither. We don’t use MailPro or mail plugin anymore.

Although I didn’t mention it as a problem, do you send mails via Graph or do you still use smtp for that part?

Hello Robert,

currently we still send with SMTP. Our focus for now was the email retrieval, as this will no longer work from October.
We have already prepared our Azure app for email sending as well. Since SMTP is still working, it will run in parallel for a while.

Hello all,

I worked on the authentication with varying degrees of desperation, and found some more things:

The very short duration when accessing seems to have another reason, there seems to be a problem when accessing many folders + the number ob emails in them, when accessing folder contents, there seems to be no problems. So I can go on fetching Mails

The other thing is, the flow is exactly as Alexander described:

get the “code” from the initial permission-request
get the access_token, expiration and a refresh_token
if you want to access account, use the refresh_token to get a new set of access_token and refresh_token.
Save the new refresh token for the next time and use the access_token as the password.

I updated the repository on GitHub - RobertEdelmann1974/office365_test to reflect that info, (and corrected the name…).

This won’t help Alexander, though, as the IMAP-listener work differently.

I didn’t find any infos on access_tokens with a longer duration, the refresh → access + new refresh is the standard. Then again, since the graph-oauth is similar to the imap-flow and graph can provide webhooks, perhaps there is a possibilty of a ybrid solution. or a move to graph…

I don’t use the MailPro plugin for IMAP, but do use the standard servoy mail plugin to send via SMTP using oAuth credentials. It took a while, and I had to view the source of the Servoy’s oAuth plugin to figure out what was going on, but I did get this working. Some snippets below. Hopefully its useful.

Some scope vars

/**
 * @type {plugins.oauth.OAuthService}
 * @private 
 */
var oAuthService = null;

Start the oAuth process. we do this onShow of the login form so it happens immediately.

function startOAuth(){
	plugins.oauth.serviceBuilder(applicationId)
		.clientSecret(clientSecret)
		.defaultScope(scope)
		.tenant(azureTenant)
		.callback(oAuthCallback, 60)
		.build(plugins.oauth.OAuthProviders.MICROSOFT_AD);
}

The oAuth callback. Note that we are saving the oAuthService in a scope variable!

/**
 * @param {Boolean} oAuthSuccess
 * @param {plugins.oauth.OAuthService} authResult
 */
function oAuthCallback(oAuthSuccess, authResult){
	if(oAuthSuccess){
		oAuthService = authResult
		
		//do whatever else you need for authentication/login, etc.
	}
	else{
		application.output("oauth error")
	}
}

Then anytime we need a token to do something like send an email, we call this. It handles getting a new token when necessary.

function getToken(){
	if(!oAuthService)
		return null
		
	if(oAuthService.isAccessTokenExpired()){
		oAuthService.refreshToken()
	}
	
	return oAuthService.getAccessToken()
}

Thanks a lot, I got it to work too after some experiments and talks with Patrick with the plugin, that makes it easier than sending manual post/get-requests to the server (and helps with integrating the first step.

(And I updated the github-repo to reflect that changes)

I’m not using the tenant-Info right now, just “common”, I’ll have to look into that.

What I’m missing right now is the handling of requests via a batch-process, I’m saving the refresh_token and use it to get the new access_token + refresh_token by hand like so (the tokens are s scope variabels)

/**
 * @properties={typeid:24,uuid:"7E5F39AE-2774-4DCB-9331-7DC6F5CBA6A0"}
 */
function refreshAccessToken() {
	if (!refreshToken) {
		return;
	}
	var httpClient = plugins.http.createNewHttpClient();
	var request = httpClient.createPostRequest('https://login.microsoftonline.com/common/oauth2/v2.0/token');
	request.addHeader('Content-Type', 'application/x-www-form-urlencoded');
	var bodyContent = 'client_id='+clientId;
	bodyContent += '&grant_type=refresh_token';
	bodyContent += '&scope='+scopeList.join(' ');
	bodyContent += '&refresh_token='+refreshToken;
	bodyContent += '&client_secret='+clientSecret;
	request.setBodyContent(bodyContent);

	var start = new Date();
	var response = request.executeRequest();
	var statusCode = response.getStatusCode()
	if (statusCode != 200) {
		application.output('Error processing request, Statuscode ' + statusCode.toString() + '\n' + response.getResponseBody());
		return;
	} else {
		/** @type {{token_type: String, scope: String, expires_in: Number, ext_expires_in: Number, access_token: String, refresh_token: String, id_token: String}} */
		var accessTokenObject = JSON.parse(response.getResponseBody());
		accessToken = accessTokenObject.access_token;
		refreshToken = accessTokenObject.refresh_token;
		accessTokenExpiresOn = new Date(start.getTime()  + accessTokenObject.expires_in*1000);
	}
}

I couldn’t find a way to get a new service object without going through a new authorization (and especially not from a headless-client that fetches mails) so I had to do it via createPostRequest().
I think it would be more elegant if you could add something like .refreshToken(refreshToken) to the plugins.oauth.serviceBuilder to skip the Authorization when the token is valid.

Another question, when i try to connect to the smtp-server to send a mail using the email-address + the access_token, I get an error that the password is invalid, how did you get the mail sent?

Glad you figured it out, and nice of you to share the examples. In my example, that code is in each users session. So the session sends emails from the logged in users Office365 account. If you are doing server-side stuff without the user logged in, there are other way to get authenticated to all accounts in the tenant (like in the case of a corporate customer where you want access to all the user’s stuff)
https://learn.microsoft.com/en-us/excha … n-requests
https://learn.microsoft.com/en-us/azure … grant-flow

Hallo Scott,

thanks for the input.

I was missing some parameters for the login / connection for SMTP, I got it to work with the mailpro plugin.

I found the missing pieces here: JavaMail, resulting in this code:

function sendSMTPMailPro() {
	var smtpAccount = plugins.MailPro.SMTPAccount('smtp.office365.com');
	smtpAccount.port = 587;
	smtpAccount.userName = '<sender address>';
	smtpAccount.requiresAuthentication = true;
	smtpAccount.useTLS = true;
	smtpAccount.password =  accessToken;
	smtpAccount.addSmtpProperty('mail.smtp.auth.mechanisms','XOAUTH2');
	smtpAccount.addSmtpProperty('mail.imap.sasl.enable', 'true');
	var connect = smtpAccount.connect();
	if (connect && smtpAccount.connected) {
		var mailMsg = smtpAccount.createMessage('<to>','<from>','<subject>','<body>');
		var success = smtpAccount.sendMessage(mailMsg);
		if (success) {
			application.output('message sent successfully.')
		}
	}
}

accessToken is a scope variable.

And I just found examples from you in an old thread. One I did look at. And still missed it.

The Servoy mail plugin works like this:

function sentSMTPServoyMail() {
	var mailProperties = new Array();
	mailProperties.push('mail.smtp.starttls.enable=true')
	mailProperties.push('mail.transport.protocol=smtp')
	mailProperties.push('security.require-ssl=true')
	mailProperties.push('mail.smtp.auth=true')
	mailProperties.push('mail.smtp.auth.mechanisms=XOAUTH2');
	mailProperties.push('mail.imap.sasl.enable=true');
	mailProperties.push('mail.smtp.auth=true');
	mailProperties.push('mail.smtp.port=587');
	mailProperties.push('mail.smtp.host=smtp.office365.com');
	mailProperties.push('mail.smtp.auth=true');
	mailProperties.push('mail.smtp.username=<username');
	mailProperties.push('mail.smtp.password='+accessToken);
	plugins.mail.sendMail('<to>','<from>','New Message','Messagebody',null,null,null,mailProperties);
}

So, I (sort of) completed the solution i put on github, I can access the user-account on Office 365 via IMAP and SMTP, if i allow a obscene amount of permissions, i needed these permissions:
[attachment=0]CleanShot 2022-10-10 at 00.47.40.png[/attachment]
But it works, at least for the personal account.
Even though I set the “Mail.Read.Shared/Mail.ReadWrite.Shared”, I can’t access shared folders, at least not in the way I can do it with basic authentication:

imapSharedName = <Name>;
props["mail.imaps.sasl.enable"] = true;
props["mail.imaps.sasl.authorizationid"] = <alias of shared account>;
imapAccount = plugins.MailPro.ImapAccount('emailaccount,
			recordEmailAccount.imap_server,
			imapUserName,
			imap_password);
rootFolder = imapAccount.connect(props)

Since there is not really much info besides “use graph” i think i give the graph method a spin, at least that way i can also access contants, calendars and tasks.

In the mean time, I hope my tests/the test-solution on ```


![CleanShot 2022-10-10 at 00.47.40.png|1031x673](upload://nc3DNI9cg7RtqAPxIxmKfELZEcW.png)

Just a quick heads up, I updated the solution under ```


One can't help but notice that Microsoft has a well structured approach to documenting Graph, like "if you want to read mails, you need that permission" + an interactive tool to experiment with the API. It feels almost like MS want's to nudge people in that direction.

Graph-Explorer is here: https://developer.microsoft.com/en-us/graph/graph-explorer
Documentation of Mail-Objects is here: https://learn.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0

robert.edelmann:
Even though I set the “Mail.Read.Shared/Mail.ReadWrite.Shared”, I can’t access shared folders, at least not in the way I can do it with basic authentication:

Ok, I found the way to access the shared accounts, just use the email-address of the shared account + the access-token of the user as name / password to log on, no extra steps required, which is nice, saves me the pressure to migrate to graph for mail under pressure.

Hi Robert,

Thanks to you (and everyone) for sharing your experience with this. I am sure it will be helpful to other developers, particularly the sample solution and the tips on how to overcome the issues you have encountered.

Thanks
Steve

Dear Robert,
also many thanks from my side to you and all other contribute to this solution.
This is a great help !!
I am still trying to make it work, but it is really good to have you sharing your knowledge.
Best regards and many thanks again
Nam

HI Robert,
i just wonder which version of Servoy are you using ?
I use Version: 2021.3.2.3644_LTS and it seems like my plugins.oauth does not have the methods (responseMode and responseType) you use in the example solution.
Best regards
Nam